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Coloured Image Assessment 



This invention relates to a method, a computer program and an apparatus for making 
measurements upon histologic* imagery to provide clinical information on potency 
cancerous tissue such as breast cancer tissue. 

Breast cancer is a common form of female cancer ones a lesion Indicative of breast 
cancer has been detected, tissue samples are taken and examined by a histopathologic 
to estabnsh a d.agnosis, prognosis and treatment plan. However, pathological analysis of 
fssue samples is a time consuming and inaccurate process. It entails Interpretation of 
colour .mages by human eye, which is highly subjective: it is characterised by 
considerable accuracies in observations of the same samples by different observers 
and even by the same observer at different times. For example, two different observers 
assessing the same ten tissue samples may easily give different opinions for three of the 
sides - 30% error. The problem is exacerbated by heterogeneity, I.e. complexity of some 
tissue sample features. Moreover, there is a shortage of pathology staff. 

Oestrogen and progesterone receptor (ER and PR) status, C-erb-2 and vascularity are 
parameters which are data of interest for assisting a ciinician to formulate a diagnosis 
prognose and treatment P ,an for a patient. C-erb-2 is aIso known as Cerb-B2 her- 2 ' 
her-2/nau and erb-2. 



It is an object of the invention to provide a technique for objective measurement of at 
20 least one of ER status, PR status, C-erb-2 and vascularity. 

in a first aspect, the present invention provides a method of measuring oestrogen or 
progesterone receptor (ER or PR) status having the steps of: 

a) obtaining histopathologic^ specimen image data; and 

b) identifying in the image data groups of contiguous pixels corresponding to 
25 respective cell nuclei; 

characterised in that the method also includes the steps oh 

c) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

d) thresholding the image data on the basis of hue and saturation and identifying 
30 pixels corresponding to cells which are preferentially stained relative to 

surrounding specimen tissue; and 
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e) determining ER or PR status from proportion of pixels corresponding to preferentially 
•stained cells. 

The invention provides the advantage that it is cornputeHmplementable, and hence is 
carried out in a way which avoids the subjectivity of a manual inspection process. 

5 In an alternative first aspect, the invention may provide a method of measuring ER or PR 
status having the steps of: 

a) obtaining hlstopattiological specimen image data; and 

b) identifying in the image data groups of contiguous pixels corresponding to respective 
cell nuclei; 

10 characterised in that the method also includes the steps oft 

c) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

d) thresholding the image data on the basis of hue and saturation and Identifying pixels 
corresponding to ceils which are preferentially stained relative to surrounding 

15 specimen tissue; and 

e) determining ER or PR status from normalised average saturation. 

, In a further alternative first aspect, the invention may provide a method of measuring ER 
or PR status having the steps of: 
a) obtaining histopathologic^ specimen image data; and 
20 b) identifying in the image data groups of contiguous pixels corresponding to respective 
cell nuclei; 

characterised in that the method also includes the steps of: 

c) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 
25 d) thresholding the image data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

e) determining ER or PR status from normalised average saturation and fraction of 
pixels corresponding to preferentially stained cells. 

30 Step b) may implemented using a K-means clustering algorithm employing a 
Mahalanobis distance metric. 
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Step c) may be Implemented by transforming the image data into a chrornaticity space, 
and deriving hue and saturation from image pixels and a reference colour. Hue may be 

obtained from an angle 0 equal to sin -1 , ^ ffi ■ and saturation from an 

expression — — where (x, y) and y) are respectively image pixel coordinates 
jc t y 

5 and reference colour coordinates in the chrornaticity space. It may be adapted to lie in 
the range 0 to 90 degrees and a hue threshold of 80 degrees may be set in step d). A 
saturation threshold S 0 may be set in step d), S a being 0.9 for saturation in the range 0,1 
to 1.9 and 0 for saturation outside this range. 

The fraction of pixels corresponding to preferentially stained cells may be determined by 
10 counting the number of pixels having both saturation greater than a saturation threshold 
and hue modulus less than a hue threshold and expressing such number as a fraction of 
a total number of pixels in the image: it may be awarded a scare 0, 1, 2, 3, 4 or 5 
according respectively to whether it is (i) 0, (ii) > 0 and < 0.01, (HI) > 0.01 and < 0.10, (iv) 
S: 0.1 1 and < 0.33, (v) £ 0.34 and < 0.66 or (vi) z 0.67 and < 1 .0. 

15 Normalised average saturation may be accorded a score 0 # 1, 2 or 3 according 
respectively to whether it is (i) £ 26%, <ii) > 25% and < 50%, (Hi) > 50% and < 75% or (iv) 
> 75% and < 100%, 

Scores for normalised average saturation and fraction of pixels corresponding to 
preferentially stained cells may be added together to provide a measurement of ER or 

20 PR. 

The method of the invention may include measuring Oerb-2 status by the following 
steps: 

a) correlating window functions of different lengths with pixel sub-groups within the 
identified contiguous pixels groups to identify pixels associated with celi boundaries, 
25 b) computing brightness-related measures of cell boundary brightness and sharpness 
and brightness extent around cell boundaries from pixels corresponding to cell 
boundaries, 

c) comparing the brightness- related measures with predetermined equivalents obtained 
from comparison images associated with different values of C-erb-2, and 
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d) assigning to the Image data a C-erb-2 value which is that associated with the 
comparison image having brightness-refated measures closest to those determined 
for the image data. 



The method of the invention may include measuring vascularity by the following steps: 
5 a) deriving hue and saturation for the image data In a colour space having a hue 
coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the image data on the basis of hue 
and saturation; 

c) identifying in the segmented image groups of contiguous pixels; and 

10 d) determining vascularity from the total area of the groups of contiguous pixels which 
are sufficiently large to correspond to vascularity, such area being expressed as a 
proportion of the image data's total area. 



In a second aspect, the invention provides a method of measuring C-erb-2 status having 
the steps of: 

15 a) obtaining histopathologic^ specimen image data; and 

b) identifying In the image data contiguous pixel groups corresponding to respective cell 
nuc\e\ associated with surrounding cell boundary staining; 

c) characterised in that the method also includes the steps of: 

d) correlating window functions of different lengths with pixel sub-groups wfthin the 
20 identified contiguous pixels groups to identify pixels associated with cell boundaries, 

e) computing brightness-related measures of ceil boundary brightness and sharpness 
and brightness extent around cell boundaries from pixels corresponding to cell 
boundaries, 

f) comparing the brightness-related measures with predetermined equivalents obtained 
25 from comparison images associated with different values of C-eib-2, and 

g) assigning to the Image data a C-erb-2 value which is that associated with the 
comparison image having brightness-related measures closest to those determined 
for the Image data. 



In this aspect, at least some of the window functions may have non-zero values of 6, 12, 
30 24 and 48 pixels respectively and zero values elsewhere. Pixels associated with a cell 
boundary are identified from a maximum correlation with a window function, the window 
function having a length which provides an estimate of cell boundary width. 
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The brightness-related measure of eel) boundary brightness end sharpness may be 
computed in step d) using a calculation inoluding dividing cefl boundaries by their 
respective widths to provide normalised boundary magnitudes, selecting a fraction of the 
normalised boundary magnitudes each greater than unselected equivalents and 
summing the normalised boundary magnitudes of the selected fraotion. 

In step d) a brightness-related measure of brightness extent around cell boundaries may 
be computed using a calculation including dividing normalised boundary magnitudes into 
different magnitude groups each associated with a respective range of magnitudes, 
providing a respective magnitude sum of normalised boundary magnitudes for each 
magnitude group, and subtracting a smaller magnitude sum from a larger magnitude 
sum. 



15 



25 



The comparison image having brightness-related measures closest to those determined 
for the Image data may be determined from a Euclidean distance between the 
brightness-related measures of the comparison image and the image data. 



In step b) identifying in the image data contiguous pixel groups corresponding to 
respeotlve cell nuclei is carried out by an adaptive thresholding technique arranged to 
maximise the number of contiguous pixel groups identified. For image data including red, 
green and blue image planes the adaptive thresholding technique may include: 

a) generating a mean value n H and a standard deviation o R for pixels in the red Image 
20 plane, 

b) generating a cyan image plane from the image data and calculating a mean value uc 
for its pixels, 

c) calculating a product CMM^ where CMM is a predeteimined multiplier. 

d) calculating a quantity R B equal to the number of adjacent linear groups of pixels of 
predetermined length and Inoluding at least one cyan pixel which is less than 

e) for each red pixel calculating a threshold equal to {RMMjlir ■ o>(4 - R B )} and RMM is a 
predetermined multiplier, 

f) forming a thresholded red image by discarding each red pixel that is greater than or 
30 equal to the threshold, 

g) determining the number of contiguous pixel groups in the thresholded red image, 

h) changing the values of RMM and CMM and iterating steps c) tog), 
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0 changing the values of RMM and CMM once more and Iterating steps c) to g). 

j) comparing the numbers of contiguous pixel groups determined in steps g) to I), 
treating the three pairs of values of RMM and CMM as points in a two dimensional* 
space, selecting the pair of values of RMM and CMM associated with the lowest 
number of contiguous pixel groups, obtaining its reflection in the fine Joining the other 
two pairs of values of RMM and CMM, using this reflection as a new pair of values of 
RMM and CMM and iterating steps c) to g) and this step j). 

The first three pairs of RMM and CMM values may be 0.602 and 1.24, 0.903 and 0.903, 
and 1.24 and 0.802 respectively. . 



Brown pixels maybe removed from, the thresholded red image if like-located pixels in the 
cyan image are less than CMMjtc; edge pixels may be removed likewise if like-located 
pixels in a Sobei-fiftered cyan image having a standard deviation o e are greater than 
(Mo* l-5o c ). Pixels corresponding to lipids may also be removed if their red green and 
blue pixel values are all greater than the sum of the relevant colour's minimum value and 
15 98% of its range of pixel values in each case, 

The thresholded red image may be subjected to a morphological closing operation. 

In a third aspect, the present invention provides a method of measuring vascularity 
having the steps of: 

a) obtaining histopathological specimen image data; 

20 characterised in that the method also includes the steps of: 

b) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

c) producing a segmented image by thresholding the image data on the basis of hue 
and saturation; and 

25 d) identifying in the segmented image groups of contiguous pixels; and 

e) determining vascularity from the total area of the groups of contiguous pixels which 
are sufficiently large to correspond to vascularity, such area being expressed as a 
proportion of the image data's total area. 



tu0480B2^25aSej^O^;2f^;q 
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In thte aspect the image data may comprise pixels with red, green and blue values 
designated R, Q and B respectively, characterised in that a respective saturation value S 
is derived in step b) for each pixel by: 

a) defining M and m for each pixel as respectively the maximum and minimum of R G 
and B; and 

b) setting S to zero if m equals zero and setting S to (M - m)/M otherwise. 
Hue values designated H may be derived by; 

a) defining new values newr, newg and newb for each pixel given by newr = (M - RV( M 
- m), newg = (M- G>/(M - m) and newb - (M - B)/(M - m) in order to convert each 
p«el value into the difference between its magnitude and that of the maximum of the 
three colour magnitudes of that pixel, this difference being divided by the difference 
between the maximum and minimum of R, G and B, and 

b) calculating H as tabulated immediately below: 



M 


H 


0 


180 


R 


60(newb - newg)* 


G 


60(2 + newr -newb)* 


B 


60(4 + newg -newr)* 



20 



provided that if H proves to be >360. then 360 Is subtracted 
from ft, and if H proves to be <0, 3S0 Is added to it 

The step of producing a 9sgmented ima9 , may bQ lmp , emenfed fay 

further processing only these pixels having both a hue H In the range 282-356 and a 

saturation S in the range 0.2 to 0 24 The <*ten rrf m^ma*-. i «. 

a * v lo Ine of identifying in the segmented image 

groups of contiguous pixels may include the step of spatially filtering such groups to 
remove groups having insufficient pixels to contribute to vascularity. The step of 
determmmg vascularity may indude treating vascularity as having a high or a low value 
according to whether or not it is at least 31 %, 
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In a fourth aspect, the present inventfon proves a computer program for measuring ER 
or PR status, the program being arranged to control computer apparatus to execute the 
steps of: 

a) processing histopathologic specimen image data to identify (n the Image data 
groups, of contiguous pixels corresponding to respective cell nuclei; 

characterised In that the program is also arranged to implement the steps of: 

b) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

c) thresholding the imaga data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

d) determining ER or PR status from proportion of pixels corresponding to preferentially 
stained cells. 

In an alternative fourth aspect, the present invention provides a computer program for 
measuring ER or PR status, the program being arranged to control computer apparatus 
to execute the steps of: 

a) processing histopathological specimen image data to identify In the image data 
groups of contiguous pixels corresponding to respective cell nuclei; 

b) characterised in that the program is also arranged to Implement the steps of 

o) denving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

d) thresholding the Image data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

e) deteimining ER or PR status from normalised average saturation. 

in a further alternative fourth aspect, the present invention provides a computer program 
for measuring ER or PR status, the program being arranged to control computer 
apparatus to execute the steps of: 

a) processing histopathologic*! specimen image data to identify in the image data 
groups of contiguous pixels corresponding to respective cell nuclei; 

characterised in that the program is also arranged to implement the steps of: 

b) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 
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c) thresholding the imag a data on the baste of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

d) determining ER or PR status from normalised average saturation and fraction of 
pixels corresponding to preferentially stained cells. 

In a fifth aspect, the present invention provides a computer program for use in measuring 
C-erb-2 status arranged to control computer apparatus to execute the steps of : 

a) processing histopathological specimen image data to identify contiguous 
pixel groups corresponding to respective cell nuclei associated with 
surrounding cell boundary staining; 

characterised in that the computer program is also arranged to implement the 
steps of: 

b) correlating window functions of different lengths with pixel sub-groups 
within the identified contiguous pixels groups to identify pixels associated 

15 with cell boundaries, 

o> computing brightness-related measures of cell boundary brightness and 
sharpness and brightness extent around ceil boundaries from pixels 
corresponding to cell boundaries, 

d) comparing the brightness-related measures with predetermined 
20 equivalents obtained from comparison images associated with different 

values of C-erb-2, and 

e) assigning to the image data a C-erb-2 value which Is that associated with 
the comparison image having brightness-related measures closest to 
those determined for the image data. 



25 



In a sixth aspect, the present invention provides a computer program for use in 
measuring vascularity arranged to control computer apparatus to execute the steps of: 

a) using histopathological specimen image data to derive hue and saturation for the 
image data in a colour space having a hue coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the image data on the baste of hue 



30 and saturation; and 



c) identifying in the segmented Image groups of contiguous pixels; and 



25-SEP-2002 12-" 27 FROM IP MALUERN 



TO UK POTENT 



P. 18/39 



10 



15 



10 

f) determining vascularity from the total area of the groups of contiguous pixels which 
are sufficiently large to correspond to vascularity, suoh area being expressed as a 
proportion of the image data's total area. 

fn a seventh aspect, the present invention provides an apparatus for measuring ER or 
PR status including means for photographing histopathologic^ specimens to provide 
image data and computer apparatus to process the image data, the computer apparatus 
being programmed to identify in the image data groups of contiguous pixels 
corresponding to respective cell nuclei, characterised in that the computer apparatus is 
also programmed to execute the steps of: 

a) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

b) thresholding the image data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

c) determining ER or PR status from proportion of pixels corresponding to preferentially 
stained cells. 

in an alternative seventh aspect, the present invention provides an apparatus for 
measuring ER or PR status including means for photographing histopathoiogfcal 
specimens to provide image data and computer apparatus to process the image data, 
the computer apparatus being programmed to identify in the image data groups of 
contiguous pixels corresponding to respective cell nuclei, characterised in that the 
computer apparatus is also programmed to execute the steps of; 

a) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

b) thresholding the image data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

c) determining ER or PR status from normalised average saturation. 

In a futher alternative seventh aspect, the present invention provides an apparatus for 
30 measuring ER or PR status including means for photographing histopathoJogicai 
specimens to provide image data and computer apparatus to process the image data, 
the computer apparatus being programmed to identify in the image data groups of 



20 
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contiguous pixels corresponding to respective cell nuclei, characterised in that the 
computer apparatus is also programmed to execute the steps of: 
a) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

b) thresholding the image data on the basis of hue and saturation and identifying pixels 
corresponding to cells which are preferentially stained relative to surrounding 
specimen tissue; and 

c) determining ER or PR status from normalised average saturation and fraction of 
pixels corresponding to preferentially stained cells. 

In an eighth aspect, the present invention provides an apparatus for measuring C-erfa-2 
status including means for photographing histopathologic! specimsns to provide image 
data and computer apparatus to process the image data, the computer apparatus being 
programmed to identify in the image data groups of contiguous pixels corresponding to 
respective cell nuclei, characterised in that the computer apparatus is also programmed 
to execute the steps of: 

a) correlating window functions of different lengths with pixel sub-groups within the 
identified contiguous pixels groups to identify pixels associated with cell boundaries, 

b) computing brightness-related measures of celi boundary brightness and sharpness 
and brightness extent around cell boundaries from pixels corresponding to ceil 
boundaries, 

o) comparing the brightness-related measures with predetermined equivalents obtained 

from comparison images associated with different values of C-erb-2, and 
d) assigning to the Image data a C-eria-2 value which is that associated with the 
comparison image having brightness-related measures closest to those determined 
for the image data. 

In a ninth aspect, the present invention provides an apparatus for measuring vascularity 
including means for photographing histopathological specimens to provide image data 
and computer apparatus to process the image data, characterised in that the computer 
apparatus is also programmed to execute the steps of; 

a) deriving hue and saturation for the image data in a colour space having a hue 
coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the image data on the basis of hue 
35 and saturation; and 



20 
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c) identifying in the segmented image groups of contiguous pixels; and 

d) determining vascularity from the total area of the groups of contiguous pixels which 
are sufficiently large to correspond to vascularity, such area being expressed as a 
proportion of the image date's total area. 

5 The computer program and apparatus aspects of the invention may have preferred 
features corresponding to those of respective method aspects. 

In order that the invention might be more fully understood, embodiments thereof will now 
be described, by way of example only, with reference to the accompanying drawings, in 
which:- 

10 Figure 1 is a block diagram of a procedure for measuring indications of cancer to 
assist in formulating diagnosis and treatment; 

Figure 2 is a block diagram of a process for measuring ER and PR receptor status in 
the procedure of Figure 1 ; 

Figure 3 is a pseudo three dimensional view erf a red, green and blue colour space 
15 (colour cube) plotted on respective orthogonal axes; 

Figure 4 Is a transformation of Figure 3 to form a chromatfeity space; 

Figure 5 is a drawing of a chromaticity space reference system; 

Figure 6 illustrates use of pofar co-orclinates; 

Figure 7 is a block diagram of a process for measuring C-erb-2 in the procedure of 

so Figure 1; and 

Figure 8 fs a block diagram of a process for measuring vascularity in the procedure of 
Figure 1 . 

The examples to be described herein are three different Inventions which can be 
implemented separately or together, because they are all measurements which 
25 individually or collectively assist a clinician to diagnose cancer and to formulate a 
treatment programme. In descending order of Importance, thB procedures are 
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determination of oestrogen and progesterone receptor status, determination of C-erb-2 
and determination of vascularity. 

A procedure 10 for the assessment of tissue samples In the form of histopathological 
slides of potential carcinomas of the breast is shown in Figure 1 . This drawing illustrates 
5 processes whioh generate measurements of specialised kinds for use by a pathologist as 
the basis for assessing patient diagnosis, prognosis and treatment plan. 

The procedure 10 employs a database which maintains digitised image data obtained 
from histological slides as will be described later. Sections are taken (cut) from breast 
tissue samples (biopsies) and placed on respsctive slides. Slides are stained using a 
10 staining agent selected from the following depending on which parameter is to be 
determined: 

a) Immunohistochemical staining for C-erb-2 with diaminobenzidine (DAB) as 
substrate (chemical staining agent) - collectively "Cerb-DAB" - this is for 
assessing C-eife-2 gene amplification status; 

b) Oestrogen receptor (ER) with DAB as substrate (collectively "ER-DAB") for 
assessing the expression (the amount expressed or emitted) of the oestrogen 
receptors. Progesterone receptor (PR) status is investigated using chemical 
treatment giving the same colouration as in ER. 

c) Immunohistochemical staining for CD31 with fuchsin (F) as substrate for 
assessing vascularity (angiogenesis). 

In a prior art manual procedure, a clinician, places a slide under a microscope and 
examines a region of it (referred to as a tile) at magnification of x40 for indications of C- 
erb-2, ER and PR status and at x20 for vascularity. 

The present invention requires data from histological slides in a suitable form. In the 
25 present example, image data were obtained by a pathologist using Zeiss Axioskop 
microscope with a Jenoptiks Progres 3012 digital camera. Image data from each slide is 
a set of digital images obtained at a linear magnification of 40 (i.e. 40X), each image 
being an electronic equivalent of a tHe, 

To select images, a pathologist scans the microscope over a slide, and at 40X 
30 magnification selects regions (tiles) of the slide which appear to be most promising in 
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terms of an analysis to be performed. Each of these regions is then photographed using 
the microscope and digital camera referred to above, which produces for each region a 
respective digitised image in three colours, i.e. red, green and blue (R, G & B). Three 
intensity values are obtained for each pixel in a pixel array to provide an image as a 
5 combination of R, G and B Image planes. This image data is stored temporarily at 1 2 for 
later use. 

Three tiles are required for vascularity measurement at 14, and one tile for each of 
oestrogen and progesterone receptor measurement at 16 and C-erb-2 measurement at 
18. These measurements provide input to a diagnostic report at 20. 

io The prior art manual procedure for scoring C-erb-2 involves a pathologist subjectively 
and separately estimating stain intensity, stain location and relative number of cells 
associated with a feature of interest in a tissue sample. The values obtained in this way 
are combined by a pathologist to give a single measurement for use in diagnosis, 
prognosis and reaching a decision on treatment The process hereinafter described in 

15 this example replaces the prior art manual procedure with an objective procedure. 

Referring now to Figure 2, processing 15 to determine ER status will be outlined and 
then described in more detail later. It begins with a pre-processing stage 30 in which a K- 
means clustering algorithm is applied to a colour image using a Mahalanobis metric. This 
determines or cues image regions of interest for further processing by associating pixels 

20 into clusters on the basis of their having similar values of the Mahalanobis metric At 32 
the colour image is transformed into a chromattoity spaoe which includes a location of a 
reference colour. Hue and saturation are calculated at 34 for pixels In clusters cued by K- 
means clustering. The number of brown stained pixels is computed at 3$ by thresholding 
on the basis of hue and saturation. An ER status measurement is then derived at 38 from 

25 a combination of the fraction of stained pixels and average colour saturation. 

The input for the ER preprocessing stage 30 consists of raw digital data files of a single 
histopathologic^ colour image or tile. A triplet of image band values for eaoh pixel 
represents the colour of that pixel in its red, green, and blue spectral components or 
image bands. These values in each of the three image bands are in the range [0...255J, 
30 where [0,0,0] corresponds to black and [255,255,265] corresponds to white. 
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The K-means clustering algorithm 30 Is applied to the digital colour image using clusters 
and the Mahalanobis distance metric. A cluster is a natural grouping of data having 
similar values of the relevant metric, and the Mahalanobis distance metrio is a 
measurement that gives an indication of degree of closeness of data items to a cluster 
5 centre. It is necessary to have some means for locating cell nuolel as pixel groups but it 
is not essential to use four clusters or the Mahalanobis distance metrio: these have been 
found to work well in identifying groups of contiguous pixels which correspond to 
respective cell nuclei. The K-means algorithm is described by J. A. Hartigan and M. A. 
Wong, in a paper entitled 'A K-means clustering algorithm', Algorithm AS 136. Applied 
10 Statistics Journal, 1979. The Mahalanobis distance metric is described by F. Heijden, in 
'Image Based Measurement Systems - object recognition and parameter estimation'. 
John Wiley & Sons, 1994 and by R. Schalkoff, in Pattern Recognition - Statistical, 
Structural and Neural approaches', John Wiley & Sons Inc., 1992. The process 
comprises an initialisation step a) followed by oomputation of a covarianee matrix at step 
b). This leads to a likelihood calculation at step c), which effectively provides the distance 
of a pixel from a cluster centre. The procedure is as follows: 

a) Initially, cluster centres are set using 30 + (cluster number + 1) x 10 subtracted from 
the mean of the red, green and blue image bands respectively. For example the first 
cluster values would be set at mean_red - 30 + (0 + 1 ) x 10 (hence mean_red - 20). 
similarly for mean_green and mean_b!ue. The second cluster would be mean_red - 
10. mean_green - 10. and mean_blue - 10, and similarly for other clusters. Pixels 
are then assigned to clusters for later readjustment 

b) For each cluster the following computations are carried out 

i) Compute elements of the kind o-* of a covarianee matrix of the image bands Indicating 

25 the degree of variation between intensities of different colours in pixels of each cluster 
from Equation (1): 

1 ^* 

where: cr* is the ij* element of the covarianee matrix, I 
N k is the number of pixels in cluster k. ! 
30 p, and o,j are the values of pixel I in image bands i and j, 

i, j take values 1, 2. 3, which represent the red, green and blue image bands ' 
respectively. 



20 
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f£ Is the mean of all pixels in image band I belonging to cluster k, and 
is the mean of all pixels in image band j belonging to cluster k. 

ii) Calculate the determinant of the covariance matrix denoted as 

iii) Calculate the inverse of the covariance matrix denoted as . 

S o). With Index i denoting pixel number, each pixel*, Is now treated as a vector having 
three elements x la ,x^,x [i3 which are the red (x tl ), green (*,, 2 ) and blue kJpixel 
values: the red, green and blue image bands are therefore represented by second 
subscript indices 1, 2 and 3 respectively. With I ranging over all pixels in a cluster k, the 
likelihood d k (x,)of a pixel vector x t not belonging to that duster is computed from 
io Equation (2) below: 



it 

where £ and £ are as defined above, 

del tnv 



& is the mean of all pixel vectors x { in cluster k, and 
t indicates the transpose of the difference vector (x, - k ) , 

is Equation (2) is re-evaluated for the same pixel vector x, in all other clusters also. Pixel 
vector x t has the highest likelihood of belonging to a cluster (denoted k„,} for which 
d k (x.) has a minimum value i.e. {d*- ft )}; cluster k m is then the most suitable to receive 
pixel x, ; l.e. find:- 

d k "(x,)£d k (x,) forall k*k m (3) 
20 Assign pixel x t to cluster km 

d). For each cluster k: 
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Store a record of which pixels belong to cluster k as an array X k , update ft with each pixel 
vector assigned to that cluster and update the number N k of pixels in that cluster. 

Calculate the cluster centre pt * for each image band j = 1 , 2 and 3 from; 

5 Iterate steps b) to d) until convergence i.e. when no more pixels change clusters or the 
number of iterations reaches a total of 20. 

The first duster {k = 1) now corresponds to Gell nuclei and the corresponding pixel 
vectors are those which are cued as of interest for output and further processing- 
Transformation of the image at 32 from red/green/blue (RGB) to chfomaticity space. In 

10 the present example, as will be described, a reference colour is used; if necessary, this 
can be avoided using e.g. the approach of the Cerb B2 example described later. The 
chemical staining used in the present example results in brown colouration and the 
approach used here is arranged to detect that preferentially, a different staining could 
however be used, fn which case the technique would be adapted to detect a different 

15 pixel colour. 

In practice brightness is liable to vary due to variation in degree of chemical staining and 
sample thickness across a slide, as well as possible vignetting by a camera lens used to 
produce the images. In consequence in this example emphasis is placed on computing a 
measurement of hue (or colour) and saturation as described later. 



20 (a) Referring now also to Figures 3 to 6, each RGB image is transformed into a 

chromaticify space. Figure 3 shows an RGB cube 40 in which red, green and 
blue pixel values (expressed as R, G and B respectively) are normalised and 
represented as values in the range 0 to 1 . These pixel values are represented 
on red, green and blue axes 52, 54 and 56 respectively. The chromaticity 

25 space is a plane 58 for which R+G+B - 1: ft is triangular within the RGB cube 

50 and passes through the points (1 ,0,0), (0,1,0) and (0,0,1 ), 
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(b) Rgure 4 shows the axes 52. 54 and 56 and chromaticity space 58 looking 
broadly speaking along a diagonal of the RGB cube 60 from the point (1,1,1) 
(not shown) to the origin (0,0,0) now referenced O for convenience. The points 
(0,0,1), (0,1,0) and (1.0,0) in Figure 3 are now referenced J, K and L 
respectively. D is a midpoint of a straight line between J and L. Image pixel 
values from the input RGB Image are projected on to the chromaticity space 
108 and the resulting projections become data points for further processing. 
The projection calculation is as follows: 



Red green and blue pixel chromaticity values r, g and b respectively are 

JO defined as:- r = , g = 2 anc j b = ts\ 

R+G+B'* R+G+B R + G + B {} 



Perpendiculars from a point P in the chromaticity space 108 to the lines JK and 
LD meet the latter at E and G respectively. Perpendiculars from P and G to the 
plane JOK meet the latter at F and H respectively. Using Equations (5), the 
35 P°ir»t P in the triangular chromaticity space 58 may then be defined by x and y 

co-ordinates shown in Rgure 4 and given by: 

x = DE = KF=$~ and y = PE = GD = b^ (6) 

(o) In Figure 5, the chromaticity space 58 is shown with x and y co-ordinate axes 
extending from an origin Q. A reference colour denoted by a point S in the 

20 drawing Is now defined as that specified for this purpose by a clinician: it is the 

colour of that part of the image which is most positively stained (the most 
intense colour on the part of the original slide from which the image was 
taken). The reference colour's RGB components are taken from the image and 
its x and y co-ordinates are computed using Equations (6) and (6): these co- 

25 ordinates are denoted as (x, y) . 

(d) In Figure 6. a polar co-ordinate system (r,8) is now defined on the (R+G+B=1) 
plane or chromaticity space 58. The co-ordinate system origin is the centre of 
gravity G of the triangle 58. A reference direction for 8 = 0 is defined as the 
direction QS of the radius vector to the reference colour S in Figure 5. For any 
30 P° int sucn as P on the triangle defined as having co-ordinates (x,y) in the 

HSV colour space, hue H Is defined as the angle 4 between the radius vector 
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(e.g. QP) to itself and the radius vector QS to the reference colour. This is 
computed at 34 from the following expressions for 



siti0 = 



xy-xy 



V* 7 + yV* 1 + y* 



(7) 



COSffl = 



xx + yy 



(8) 



and the angle $ Is defined to be sin" 1 



(9) 



For convenience the definition of hue H Is now altered somewhat to render ell values 
positive and in the range 0 to rate: the transformation of earlier values $ into a new 
version y is shown In Table 1 below: 



TABLE 1 



Condition 


Magnitude of y (New Hue H) 


sin § > 0 and cos $ > 0 




sin <j) > 0 and cos $ < 0 


7t - (J) 


sin <)>< 0 and cos ^ > 0 




sin $ < 0 and cos 4> < 0 





10 



A hue (H) threshold ¥o is set at 36 by a user or programmer of the procedure as being 
not more than */2, a typical value which might be chosen being 80 degrees. Saturation S 
is defined to be 



saturation = f* + 3 ^ (10) 
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Two values of saturation threshold S 0 are set according to whether or not image pixel 
saturation value S lies in the range 0.1 to 1 .9: this ts set out in Table 2 below: 



TABLE 2 



Saturation S 


So 


Either S < 0.1 or S > 1 .9 


0 


0.1 £ Ss 1.9 


0,9 



5 At 36, the thresholds are used to count selectively the number N fi of pixels which are 
sufficiently brown (having a large enough value of saturation) having regard to the 
reference colour. AH H and S pixel values in the image are assessed. The conditions to 
be satisfied by a pixel's hue and saturation values for it to be counted in the brown pixel 
number N b are set out in Table 3 below. 



10 TABLE 3 



Condition 


Action 


For each pixel with both hue modulus 
\y\ < iff Q and saturation S>S (> 


Treat as a "saturated" pixel; 
increase count Nb of brown 
pixels by 1 


For each pixel with \yr\> ^ 0 and/or 
saturation $ <s Q 


Treat as an "unsaturated" 
pixel; leave N b unchanged 



The average saturation of the N b saturated pixels determined in Table 3 is computed by 
adding ail their saturation values S together and dividing the resulting sum by The 
maximum saturation value of the saturated pixels is then determined, and the average 
15 saturation is normalised by expressing it as a percentage of this maximum: this approach 
is used to counteract errors due to variation in colour staining between different images. 
The normalised average saturation is then accorded a score at 38 of 0, 1. 2 or 3 
according respectively to whether this percentage is (a) < 25%, (b) > 25% and £ 50%, (c) 
> 50% and £ 75% or (d) > 75% and <; 100%. 
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The fraction of saturated pixels - those corresponding to cells stained sufficiently brown 
relative to surrounding tissue - is computed at 38 from the ratio IVN where N is the total 
number of pixels in the image. This fraction is then quantised to a score in the range 0 to 
5 as set out in Table 5 below. 



Nb/N : Fraction of image 
pixels that axe stained 


Score 


0.00 


0 


<0.01 


1 


0.01 -0.10 


2 


0.11-0.33 


3 


0.34-0.66 


4 


0.67-1.00 


5 



5 TABLES 

The two scorns determined above, i.e. for normalised average saturation and fraction of 
sufficiently brown pixels are now added together to give a measure in the range 0 to 8. 
The higher this number Is, the more oestrogen (ER) positive the sample is, as shown in 
Table 6 befow. 



Description of ER status (ER Score) 


Range 


Strongly positive 


7-8 


Positive 


4-6 


Weakly positive 


2-3 


Negative 


0-1 



TABLE 6 



15 Women w?th en ER score of 7 or 8 will respond favourably to hormonal treatment such 
as Tamoxifen; women with an ER score in the range 4 to 6 will have 50% of chance of 
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responding to this treatment. Women scoring 2 or 3 will not respond v B ry wall, and those 
scoring 0 or 1 will not respond to hormonal treatment at all. 

Images for ER and PR are indistinguishable visually and they are distinguished by the 
fact that they are produced using different stains. A PR score is therefore produced from 
5 stained slides in the same way as an ER score described above. The significance of 
• progesterone receptor (PR) positivlty in a breast carcinoma is less well understood than 
the equivalent for ER. In general, cancers that are ER positive will also be PR positive. 
However, carcinomas that are PR positive, but not ER positive, may have a worse 
prognosis. 

io Turning now to C-eib-2 the conventional manual technique Involves processing a 
histopathological slide with chemicals to stain It appropriately, after which it is viewed by 
a clinician. Breast cells on the slide will have stained nuclei with a range of areas which 
allows discrimination between tissue cells of interest and unwanted cell types which are 
not important to cancer assessment Cancerous cells will usually have a lanjer range of 

15 sizes of nuclei which must be allowed for in the discrimination process. A clinician needs 
to ignore unwanted cell -types and to make a measurement by subjectively grading cells 
of interest as follows: 



Score 


Staining Pattern 


0 


membrane staining in less than 1 0% of cells 


1 


just perceptible membrane staining in more than 10% of cells but 
membranes incompletely stained 


2 


weak to moderate complete membrane staining of more than 1 0% of cells 


3 


strong complete membrane staining of more than 10% cells 



Scores 0 and 1 are negative (not justifying treatment), whereas scores 2 and 3 are called 
20 positive Gustrfying treatment). 

Unfortunately, there are artefacts which make measurement more complicated, as 
follows: 

Retraction (shrinking) artefact: less sharply defined than true membrane staining; 
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Thermal artefact if a electrocautery instrument is used, rather ill-defined staining 
occurs; 

Crushing artefact, the tissue Is inadvertently mechanically deformed allowing 
more ill-defined staining. 

5 Thermal and crushing artefacts are normally confined to boundaries of a tissue specimen 
and would hopefully be excluded to some extent by a clinician photographing tiles from a 
slide. However, it is still important to guard against ill-defined staining not attached to a 
cell membrane. 

10 The technique of this invention attempts to measure the parameters mentioned above 
namely: 

Completeness of cell membrane staining; 

Intensity and thinness of cell membrane staining; and 

Ratio of cell membrane staining. 

15 There are two main stages in the present invention, and these may optionally be 
preceded by pre-processing if images are poor. The main stages are: 

finding cell nuclei which satisfy area and location limitations associated with 
tumours; and 

determining a score which characterises the membranes of the cell nuclei found in 
20 the preceding stage. 

Referring now to Figure 7, the C-ero-2 technique of the invention will firstly be outlined 
and later described in more detair. An optional preprocessing step 70 is carried out if 
images of tiles are poor due to camera vignetting or colour errors across the image. 

image segmentation is carried out In steps 71 to 78, i.e. automated separation of objects 
25 from a background in a digital image. The original digital image of a tile has red. green 
and blue image planes: from the green and blue image planes a cyan image plane is 
derived at 71 and a SobeMiitered cyan image plane at 72. There are now five image 
planes: of these only the red and blue Image planes are essential with conventional 
colour staining, the other image planes are used for desirable but not essential filtering 
30 operations upon the red image planes. Statistical measures of the five Image planes are 
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computed at 74 and 76, and then a segmented image is optimised and generated at 78 
which has been filtered to remove unwanted pixels and spatial noise. The segmented 
Image identifies cell nuotei. Step 78 is an adaptive thresholding technique using 
information from regions around pixels: it is shown in more detafl within chain lines 80 
a with arrows 82 indicating iterations. It is an alternative to the K-means clustering 
algorithm previously described, which could also be used. 

If at 84 the number of ceils found is Jess than 16, the image is rejected at 86: if if is 16 or 
greater, then having found the cell nuclei, and hence the cells, the strength, thinness and 
completeness of each cell's surrounding membrane staining are measured and the 
10 membrane stainings are then ranked. 

For each cell, at 88 a sequence of cross-correlation windows of varying widths is passed 
along four radii from the cell centroid to determine the cell boundary brightness value, 
membrane width and distance from the centrold of the most intense staining. Cell 
boundary brightness value is normalised by dividing by membrane width, and nuclear 

15 area and sum of normalised boundary brightness values are then obtained. Statistical 
measures characterising membrane-staining strength, specificity and completeness are 
then deduced: these measures are compared with equivalents obtained from four 
reference images. The measured image is then graded by assigning it a score which is 
that of the closest reference, with the metric of Euclidean-distance. Other metrios may 

20 also be used. Alternatively, the scores of a moderately large sample may be used as 
references. 



The C-erb-2 process will now be described in more detail. The process 18 is applied to 
one image or tile obtained by magnifying by a factor of 40 an area of a histological slide. 
Referring to Figure 7 once more, The optional preprocessing step 70 is carried out by 
25 either 

(a) dividing the image into a suitable number of tiles (with less Individual variability) 
and processing them separately - this should be considered an option in general, 
though it is not necessary if there is reasonable uniformity across individual 
images; or 

30 (b) preferably, if sufficient images are available from the same camera objective 

lens, computing its deficiency and correcting it, rather than processing sub-images 
with more part-cells split across boundaries. 
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The digital image of a slide is a three colour or red green and blue (RGB) image as 
defined above, i.e. there is a respective image plane for each colour. For the purposes of 
the following analysis, the letters R. G and B for each pixel are treated as the red green 
and blue intensities at that pixel. The* RGB image is used at 71 to oompute a cyan image 
s derived from the blue and green image planes: i.e. for each pixel a cyan Intensity C is 
oomputed from C = (2xB + G)/3. the respective pixel's green (G) intensity being added 
to twice its blue (B) Intensity and the resulting sum being divided by three. Whan 
repeated for ail pixels this yields a cyan image or image plane. Cyan Is used because it 
is a complementary colour to brown, which is the cell boundary colour produced by 

10 conventional chemical staining of a specimen. The blue image plane could be used 
instead but does not normally produce results as good as the cyan image. If a different 
colour staining were to be use. the associated complementary colour image wouid be 
selected. This process step Is not essential but it greatly assists filtering out unwanted 
pixels and it does so without a reference colour (see the ER/PR example which uses an 

15 alternative approach). 



At 72, a Sobei edge filter is applied to the cyan image plane: this is a standard Image 
processing technique published In Kiette R., & Zamperonl P., 'Handbook of Image 
processing operators', John Wiley & Sons. 1 996. A Sobel edge filter consists of two 3x3 
arrays of numbers S P and S Q , each of which is convolved with successive 3x3 arrays of 
20 pixels in an image. Here 





"1 
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1 " 
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and Sq = 


2 


0 


-2 




-1 


-2 


-1_ 




I 


0 


-1_ 



The step 72 initially selects a first cyan 3x3 array of pixels in the top left hand corner of 
the cyan image: designating as Q,a general cyan pixel in row i and column J, the top left 
hand corner of the image consists of pixels ft, to ft 3 , Cat to and C 31 to C^. c, is 
25 then multiplied by the respective digit of Sp located in the S P array as q, is in the 3x3 
cyan pixel array: Le. ft, to C, 3 are multiplied by 1, 2 and 1 respectively, C 21 to C23 by 
zeroes and Ca, to by -1, -2 and -1 respectively. The products so formed are added 
algebraically and provide a value p. 
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The value of p will be relatively low for pixel values changing slowly between the first and 
third rows either side of the row of C22, and relatively high for pixel values changing 
rapidly between those rowst in consequence p provides an indication of Image edge 
sharpness across rows. This procedure is repeated using the same pixel array but with 
5 So replacing Sp, and a value q is obtained: q Es relatively low for pixel values changing 
slowly between the first and thirci columns either side of the column of C22, and relatively 
high for pixel values changing rapidly between those columns: and q therefore provides 
an indication of image edge sharpness across columns. The square root of the sum of 

the squares of p and q are then computed i.e.^/p 2 +q 2 , which is defined as an "edge 
10 magnitude" and becomes T^ (replacing pixel Cv> at the centre of the 3x3 array) in the 
transformed cyan image. It is also possible to derive an edge "phase angle" as tan -1 p/c?r 
but that is not required in the present example. 

A general pixel Ty {row I, column j) in the transformed image is derived from Cmj-i to C h 
ij+i, Cij.-i to Qj + i and C^-,^ to of the cyan image. Because the central row and 

IS column of the Sobel filters in Equation (11) respectively are zeros, and other coefficients 
are 1 s and 2s, p and q for T s can be calculated as follows: 

. p - { c Hm h + 2c Mil + a lMfM } - { + 2 c M rl + Cm,!*} (1 2) 

q = { Cm + 2C, rH + Cw fiM } - { C MiJ+1 + 2C w + C k1Jrt } (13) 

Beginning with i=j-2 r p and q are calculated for successive 3X3 pixel arrays by 
20 incrementing j by 1 and evaluating Equations (2) and (3) for each such array until the end 
of a row Is reached; j is then incremented by 1 and the procedure is repeated for a 
second row and so on until the whole image has been transformed. This transformed 
image is referred to below as the "Sobel of Cyan" image or image plane. 

The Sobel filter cannot calculate values for pixels at image edges having no adjacent 
25 pixels on one or other of its sides: i.e. in a pixel array having N rows and M columns, 
edge pixels are the top and bottom rows and the first and last columns, or in the 
transformed image pixels Tn to T™, T w to T™, T„ to T 1M and T 1M to T WM . By convention 
in Sobel filtering these edge pixels are set to zero. 
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A major problem with measurements on histopathologicai images is that the staining of 
different slides can vary enormously, e.g. from blue wfth dark spots to off-white with 
brown outlines. The situation can be improved by sifting, the slides and using only those 
that conform to a predetermined colouration. However, it has been found that it is 
s possible to cope with variation in staining to a reasonable extent by using statistical 
techniques to normalise images; m this connection steps 74 and 76 derive a variety of 
statistical parameters for use int image segmentation in step 78. 

In Step 74 Is computed the mean and standard deviation of the transformed pixel values 
TV For convenience a change of nomenclature is implemented: index k is substituted for i 
10 and J, i.e. k- 1 to NM for i, j = 1, i to N, M: this treats a two dimensional image as a 
single composite line composed of successive rows of the [mage. Also x is substttuted 
for T in each pixel value, so Tg becomes x*. The following Equations (14) and (15) 
respectively are used for computing the mean n and standard deviation or of the 
transformed pixels x^. 



15 *-j5rf*' (14) 



At 76, various statisticaF parameters are computed for the Red, Green, Blue and Cyan 
image planes using Equations (14) and (1 5) above. 

For the Red image plane the statistical parameters ^re the mean ji R and standard 
20 deviation cr R of its pixei values: in Equations (14) and (15), represents a general pixel 
value in the Red image plane. In addition, the Red image plane's pixels are compared 
with one another to obtain their maximum, minimum and range (maximum - minimum). 
Similarly, pixels in each of the Green and Blue image planes are compared with one 
another to obtain a respective maximum, minimum and range for each plane. Finally, for 
25 the Cyan image, pixels' mean and standard deviation are computed using Equations (14) 
and (15), in which x k represents a general pixel value in the Cyan Image plane. 
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In step 78, the image is segmented to identify and locate cell nuclei, a pixel is counted as 
part of a cell nucleus if and only If it survives a combination of thresholding operations on 
the Red, Green, Blue, Cyan and Sobel of Cyan image planes followed by closure of 
image gaps left after thresholding operations. It is necessary to determine .threshold 
values in a way which allows for variation in chemical staining between different images. 
The technique employed In this example is to perform a multidimensional optimisation of 
some thresholds with nuclei-number as the objective-function to be maximised: i.e. for a 
given image, threshold values are altered intelligently until a near maximum number of 
nuclei is obtained. Starting values are computed for the optimisation routines by 
choosing those suitable for provision of threshold levels. In this example, two 
dimensional optimisation is used requiring three starting values indicated by suffixes 1, 2 
and 3 and each with two components: the starting values represent vertices of a triangle 
in a two dimensional plane. The starting values are RMM1/CMM1, MMM2/CMM2 and 
RMM3/CMM3, RMM indicating a "Red Mean Multiplier" and CMM indicating a "Cyan 
15 Mean Multiplier*. Tests using a substantial number of images have shown that suitable 
starting values are RMM1 = 0.802, CMM1 = 1.24, RMM2 = CMM2 » 0.903, RMM3 = 
1.24 and CMM3 = 0.802. 

For images counterstained with Haemotoxylin and Eosln (H&E) cell nuclei are strongly 
stained blue - Le. they have very low values in the complementary ied plane. Hence the 
20 red plane is the primary plane used in thresholding as follows: 

(a) Produce a threshofded image for the Red Image plane (approximately 
complimentary to Blue) as follows: for every Red pixel value that is less than an 
adaptive threshold, set the corresponding pixel location in the thresholded Red 
image to 1, otherwise set the latter to 0. A respective adaptive threshold is 
computed separately for every pixel location as follows. At a) in step 78, the Red 
image threshold value Is dependent on the presence of enclosing brown stain In 
the neighbourhood of each pixel, i.e. it is a function of Cyan mean pc and Red 
mean p*. A check for enclosing brown is performed by searching radially outwards 
from a pixel under consideration. The procedure Is in the Cyan image plane to 
select the same pixel location as in the Red image plane and from it to searoh in 
four directions - north, south, east and west directions - for a distance of seventy 
pixels (of as many as are available up to seventy). Here north, south, east and 
west have the following meanings: north: upward from the pixal in the same 
column; south: downward from the pixel in the same column; east; rightward from 



25 



30 



|qMaQ82f25,^Sep7i02Hft2;i29i 



25-SEP-2002 

* 



5 



10 



15 



20 



. 25 



12:32 FROM IP MALUERN T0 UK PATENT 



P. 37/39 



29 

the pixel in the same row; and west: leftward from the pixel in the same row. More 
directions (e.g. diagonals north-east, north-west, south-east and south-west) could 
be used to improve accuracy but four have been found to be adequate for the 
present example, in any of these directions or radii either a cyan pixel will fall 
below a threshold (indicating a brown pixel) or a radius of 70 pixels will be reached 
without a cyan pixel doing so. The number R B of "brown" radii (radii Intersecting at 
least one brown pixel) is then used to change the red threshold adaptiveiy in the 
following way: There Is calculated a new Red image plane threshold 
RTN = RMMImr - o R (4 - R B ), where RMM1p„ Is the produot of RMMIand p* and 
o R is the standard deviation of the Red image plane. A limit is placed on RTN 
giving it a maximum possible value of 255, If the Red Image plane pixel under 
consideration is less than the Red image plane threshold calculated for it. the 
corresponding pixel at the same location in the thresholded Red image Is set to 
one, otherwise it is set to zero. 

(b) Using the Cyan Image plane, and with the Cyan mean pc from step 74, for 
every Cyan pixel value that is less than the product of CMM1 and nc, set the pixel 
in the corresponding location in the thresholded Red image to 0, otherwise do not 
change the pixel. This has the effect of removing excess brown pixels. 

(c) Using the Sobel of Cyan image plane, and with the Cyan mean Uc and 
standard deviation er c from step 74: i.e. for every Cyan pixel value that is greater 
than (p c + 1.5oc) set the corresponding pixel in the thresholded Red image to 0, 
otherwise do not change the pixel. This has the effect of removing brown edge 
pixels. 

(d) Pixels corresponding to lipids are now removed as follows: using the pixel 
minimum and range values computed at step 76, a thresholded Red image is 
produced using data obtained from the Red, Green and Blue Image planes: for 
each Red, Green and Blue pixel group at a respective pixel location that satisfies 
ail three criteria at (i) to (Hi) below, set the pixel at the corresponding location in the 
thresholded Red Image to 0, otherwise do not change the pixel; this has the effect 
of removing lipid image regions (regions of fat whioh appear as highly saturated 
white areas): Removal of these regions is not essential but is desirable to improve 
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processing. The criteria for each set of Red, Green and Blue values at a respective 
pixel are: 

(0 Red value > Red minimum + 0.98 x {red range), AND 
(H) Green pixel > Green minimum + 0.98 x (green range), AND 
(iff) Blue pixel > Blue minimum + 0.B8 x (blue range) 

Steps (e) and (d) could be moved outside the recursion loop defined within chain 
lines 80 if desired, with consequent changes to the procedure. 

(e) The next step is to apply to the binary image obtained at step (d) of 78 above a 
morphological closing operation, which consist of a dilation operation followed by 
an erosion operation. These morphological operations fuse narrow gaps and 
eliminate small holes in Individual groups of contiguous pixels appearing as blobs 
in an image. They are not essential but they improve processing. They can be 
thought of as removal of irregularities: or spatial "noise", and they are standaixJ 
Image processing procedures published in Umbaugh S.C., 'Colour vision and 

15 image processing", Prentice Hall, 1 998. 

(f) A connected component labelling process is now applied to the binary image 
produced at step (e). This is a known image processing technique (sometimes 
referred to as iDlob colouring') published by R Klette and P Zamperoniu, 'Handbook 
of Image Processing Operators', John Wiley & Sons, 1996. and A Rosenfeld and A 
C Kak, 'Digital Picture Processing', Vols.. 1 & 2, Academic Press, New York, 1982. 
It gives numerioal labels to "blobs" in the binary image, blobs being regions or 
groups of like-valued contiguous or connected pixels in an image: i.e. each group 
or blob consists of connected pixels which are all 1s, and each is assigned a 
number different to those of other groups. This enables individual blobs to be 

25 distinguished from others by means of their labels. The number of labelled image 

regions or blobs in the image is computed from the labels and output. Connected 
component labelling also determines each labelled image region's centroid (pixel 
location of region centre), height, width and area. Image regions are now removed 
from the binary image if they are not of interest because they are too small or too 

30 large in area or they have sufficiently dissimilar height and width indicating they are 
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flattened. The remaining regions in the binary image pass to the next stage of 
processing at (g). 

Steps (a) to (f) are carried out for all three starting points or triangle vertices 
RMM1/CMM1, RMM2/CMM2 and RMM3/CMM3: this yields three values for the 
s number of regions remaining In the binary image in each case. 

(g) This step is referred to as the Downhill Simplex method: it is a standard iterative 
statistical technique for multidimensional optimisation published in Nelder J.A., 
Mead R., 1966, Computer Journal, vol. 7, pp 308-313. 1965. It takes as input the 
three numbers of regions remaining after step (f). It is possible to use other 

10 optimisation techniques such as that referred to as Powelf which uses gradients. 

The starting pointfvertex yielding the lowest number of regions remaining is then 
selected. A new starting point is then generated as the reflection of the selected 
vertex in the line joining to the two other vertices; i.e. if the three vertices were to 
have been at 1,1, 1,2 and 2,1, and 1,1 was the selected vertex, then the new 

is starting point is 2,2. The selected vertex Is then discarded and the other two 

retained. The new starting point or vertex becomes RMM4/CMM4 and steps (a) to 
(f) are repeated using it to generate a new number of regions remaining for 
comparison with those associated with the two retained vertices. Again a vertex 
yielding the lowest number of regions remaining is selected, and the process of 

20 new RMM/CMM values and steps (a) to (f) is iterated as indicated by arrows 82. 

Iterations continue until the rate of change of remaining number of image regions 
(cell nuclei number) slows down, i.e. when successive iterations show a change of 
less than 10% in this number: at that point optimisation is terminated and the 
binary image remaining after step (f) selected for further processing is that 

25 - generated using the RMM/CMM values giving the highest nuclei number. 

The procedure 18 is now concerned with determining quantities referred to as 
"grand_mean" and "mean_range" to be defined later. If the Downhill Simplex method (g) 
has determined that there are less than a user specified number of image regions or cell 
nuclei, sixteen in the present example, then at 84 processing is switohed to 86 indicating 
30 a problem image which is to be refected. 
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if the Downhill Simplex method has determined that there are at least sixteen image 
regions, then at 84 processing is switched to 88 where a search to characterise these 
regions' boundaries is carried out. The search uses each region's area and centroid pixel 
location as obtained in connected component labelling at 73(f), and each region is 
5 assumed to be a cell with a centroid which is the centre of the cell's nucleus. This 
assumption Is justified for most cells, but there may be misshapen cells for which it does 
not hold: it is possible to discard misshapen cells by eliminating those with concave 
boundary regions for example, but this is not Implemented in the present example. 

The search to characterise the regions' boundaries is carried out along the respective 
10 north, south, east and west directions (as defined earlier) from the centroid (more 
directions may be used to improve accuracy): it is carried out in each of these directions 
for a distance 8 which is either 140 pixels or 2^ region area , whichever is the lesser. It 
employs the original (2B+G)/3 cyan image because experience shows that this image 
gives the best defined cell boundaries with the slide staining, previously described. 
15 Designating Qj as the intensity of a region's centroid pixel in the cyan image at row i and 
column j, then pixels to be searched north, south, east and west of this oentroid will have 
intensities in the cyan Image of C MJ to C MJ to Cwj. C 1J+1 to C,, !+8 and C I(H to C,j« 
respectively. The cyan intensity of eaoh of the pixels to be searched is subtracted from 
the centroid pixel's cyan intensity Gj to produce a difference value, which may be positive 
!0 or negative. In a cyan image, a cell nucleus is normally blue whereas a boundary is 
brown (With Staining as described earlier). 



Each pixel is then treated as being part of four linear groups or "windows" of six, twelve, 
twenty-four and forty-eight pixels each including the pixel and extending from it in a 
continuous line north, south, east or west (as defined earlier) according respectively to 

2S whether the pixel is north, south, east or west of the centroid. In effect pixels in each of 
the chosen directions have mathematical window funotions applied to them, the function 
having the value 1 at pixels within a group and the value 0 outside it. In the linear groups 
in the present example, C,«,, is for example grouped with C,«j to C l+6<i , to CW C^j 
to Ci +24J , and to Gmbj (inclusive in each case). This provides a total of 165 groups 

30 from 45 groups in each of four directions. For each group the difference between each of 
its pixels' cyan intensities and that of the centroid is calculated: the differences are 
summed over the group algebraically (positive and negative differences cancelling one 
another). This sum is divided by the number of pixels in the group to provide a net 
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difference per pixel between the cyan intensities of the group's pixels and that of the 
centrold. 

For each direction, i.e. north, south, east and west, there is now a respective set of 46 
net differences per pixel: in each set the net differences per pixel are compared and their 
maximum value is identified. This produoes a respective maximum net difference per 
pixel for each of the sets, i.e. for each of the north, south, east and west directions, and 
size of window {number of pixels in group) in which the respective maximum occurred. 
The four maxima so obtained (one for each direction) and the respective window size in 
each case are stored. Each maximum is a measure of the region boundary (cell 
membrane) magnitude in the relevant direction, because in a cyan image the maximum 
difference as compared to a blue cell nucleus occura at a brown cell boundary. The 
window size associated with each maximum indicates the region boundary width, 
because a boundary width will give a higher maximum in this technique with a window 
size whtoh it more nearly matches as compared to one it matches less well. Greater 
15 accuracy is obtainable by using more window sizes and windows matched to cell 
boundary shape, i.e. multiplying pixels in each linear group by respective values 
collectively forming a boundary shape function. The process is in fact mathematically a 
correlation operation in which a window shape is correlated with a linear group of pixels. 
A further option is to record the position of the maximum or boundary (ceil radius) as 
20 being that of one of the two pixels at the centre of the window in which the maximum 
occurs: this was not done In the present example, although it would enable misshapen 
cells to be detected and discarded as being indicated by significant differences in the 
positions of maxima In the four directions, and it would improve width measure by 
accounting for oblique intersections of windows and cell boundaries- 



25 



Each maximum or region boundary magnitude is then divided by the associated window 
size (region boundary width) used to derive it: this forms what is oalled for the purposes 
of this specification a normalised boundary magnitude - it Is a measure of both 
brightness and sharpness: It enables discrimination against ill-defined staining not 
attached to a cell membrane. 

30 The next step 90 is to apply what is referred to as a "quicksort" to the four normalised 
boundary magnitudes to sort them into descending ordsr of magnitude. Quicksort is a 
known technique published by Klette FL, Zamperoniu P., 'Handbook of Image Processing 
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Operators', John Wiley & Sons, 1996, and will not be described. It Is not essential but 
convenient. For each image region, measurements made as described above are now 
recorded In a respective 1 -dimensional vector as set out in Table 7 below: in this table 
the directions North, East etc are lost in the quicksort ordering into largest, second 
5 largest, third largest and smallest. 



TABLE 7 



Item number 1 


Parameter 


1 


Largest normalised boundary magnitude 


2 


Second Largest normalised boundary magnitude 


3 


Third Largest normalised boundary magnitude 


4 


Smallest normalised boundary magnitude 


5 


Sum of Largest, Second Largest, Third Largest and 
Smallest normalised boundary magnitudes 



A further quicksort is now applied (also at 90) to the image regions to sort them into 
descending order of ttem 6 values in Table 7 above, i.e. sum of Largest, Second Largest, 
Third Largest and Smallest normalised boundary magnitudes. A subset of the image 
10 regions is now selected as being those having large values of item 5; these are the most 
significant image regions and they are the best one eighth of the total number of image 
regions in terms of item 5 magnitude. From this subset of image regions the following 
parameters are computed at 92, "grand_mean", "mean^nge" and "relative_range" as 
defined befow : 



15 ootile = one eighth of the total number of image regions or cell nuclei (16) 

boundaries = normalised boundary magnitudes (17) 

2 a sum of . . . (over ail boundaries in the subset or best octlie) (1 8) 

item 1 = Largest normalised boundary magnitude (19) 

item 3 = Third Largest normalised boundary magnitude (20) 
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grancLmean = 6x[(£ Largest boundaries) + (£ Second Largest boundaries) 

+ (E Third Largest boundaries) + (£ Smallest boundaries)]/ 4octile 
mean_range = [(E item 1) - (2 tern 3)]/ootHe 
relative_range = 10xmeanj'ange/grand._mean 

5 Grandjnean is indicative of the degree to which an Image exhibits good ooll boundary 
sharpness and brightness. Relative_jange indicates the degree to which an Image 
exhibits brightness extending around cell boundaries - the smallest boundaries (item 4) 
are omitted from this computation to provide some robustness against incomplete cells. 
A cell boundary that exhibits a large value of refative_range will have brightness varying 
10 appreciably around the boundary corresponding to non-uniformity of staining or possibly 
even absence of a boundary. 

At 04 an overall distance measure Is computed: this measure provides an estimate of 
how far the current cyan image (generated at 71) is from each member of a 
predetermined standard set of images, four images In the present example. In this 
example the distance measure is computed against a set of four predetermined standard 
images: the standard images were obtained by dividing a large test dataset of images 
into four different image types corresponding respectively to four different C-erb-2 status 
indicators (as will be described later in more detail). The images of each image type 
were analysed to determine grand mean and relative range for each image using the 
process 18. A respective average grand mean M r {i = 0, 1, 2 and 3) and a respective 
average relative range RRj were determined for the images of each of the four Image 
types. As an alternative, it is also possible to select four good quality images of the 
relevant types by inspection from many images, and to determine Mi and RR, from them. 
The values M f and. RR f become the components of respective four-element vectors M 
and RR , and are used in the following expression: 

C-erb-2 indicator - mx{(M f - grand mean) 2 +{RR t -relative range)*} (24) 

where mini is the value of f ( i s 0, 1 , 2 or 3) for which the expression wtthin curved 
brackets { } on the right of Equation (24) is a minimum. For the vector M, from the 
dataset the following elements were determined: M Q ~1 2.32, = 23.1 6, M 2 *r 42,34 and 
30 M 3 = 87.35; elements determined likewise for the vector RR were RR 0 - 2.501, 
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RR 1 » 1.85, RR a = 1 .1 11 and RRa = 0.5394. The value of the index i is returned as the 
indicator for the Oerb-2 measurement process. 

If a value of T = 3 is obtained in the C-erb-2 measurement process, this is regarded as a 
strongly positive result: the patient from whom the original tissue samples were taken is 
5 regarded as highly suitable for treatment, currently with herceptin. A value of i = 2 is 
weaMy positive indicating doubtful suitability for treatment, and 1 = 1 or 0 is a negative 
result indicating unsuitabillty. This is tabulated below in Table 8. 

TABLE 8 



C-erb-2 status 


i Value 


Strongly positive 


3 


. Weakly positive 


2 


Negative 


0,1 



10 Referring now to Figure 8 9 there is shown a flow diagram of the process 14 (see Figure 
1) for measurement of vascularity. The process 14 is applied to three images each of x20 
magnfflcation compared to the histopathological slide from which they were taken. At 
10D each image is transformed from red/green/blue (RGB) to a different image space 
hue/saturation/value (HSV), The RGB to HSV transformation Is described by K. Jack in 
IS Video Demystified', 2 nd ed. r HighText Publications, San Diego, 1 996, In practice value V 
(or brightness) is liable to vary due to staining and thickness variations across a slide, as 
well as possible vignetting by a camera lens used to produce the images, tn 
consequence In this example the V component is ignored; It is not calculated, and 
emphasis Is placed on the hue (or colour) and saturation values H and S. H and S are 

20 calculated for each pixel of the two RGB images as follows: 

Let M = maximum of (R,G,B) (25) 

Let m =s minimum of (R.G.B) (26) 

Then newr = (M - R)/(M - m) (27) 

newg = (M — G)/(M - m) and (28) 
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newb = (M- B)/(M - m) (29) 

This converts each colour of a pixel into the differenoe between its magnitude and that of 
the maximum of the three colour magnitudes of that pixel, this difference being divided 
by the difference between the maximum and minimum of (R.G.B). 

Saturation (S) is set as follows: 

If M equals zero, thenS = 0 (30) 

if M does not equal zero, then S - (M - m)/M (31) 

The calculation for Hue (H) is as follows: from Equation (25) M must be equal to at least 
one of R, Q and B: 

if M equals zero, then H =180 (32) 

If M equals R then H = 60(newb - newg) (33) 

If M equals G then H = 60(2+ newr-newb) (34) 

If M equals B then H = 60(4 + newg - newr) (35) 

If H is greater than or equal 360 then H = H - 360 (36) 

If H is less than 0 then H = H + 360 (37) 

The Value V is not used in this example, but were it to be used it would be set to the 
maximum of (R,Q,B). 



The next step 102 is to apply colour segmentation to obtain a binary image. This 
segmentation is based on thresholding using the Hue and Saturation from the HSV 
20 colour space, and is shown in Table 9 below. 

TABLE 9 



Threshold Criterion 


Binary Image Pixel Value 


Pixel with both Hue H in the range 282 - 356 degrees 
(scale 0 to 360), and Saturation S in the range 0.2 to 
0.24 (scale 0 to 1) 


Set pixel to 1 


Pixel with either Hue outside the range 282 - 356 
degrees, and/or Saturation outside the range 0.2 - 0.24 


Set pixel to 0 
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This produces a segmented binary image in which pixels set to 1 are processed further 
and those set to 0 are discarded. 



The next stage 104 is to apply connected component labelling (as defined previously) to 
the segmented binary image; this provides a binary image with regions of contiguous 
5 pixels equal to 1, the regions being uniquely labelled for further processing and their 
areas being determined. The labelled binary Image is then spatially Uttered to remove 
small connected components (image regions with less than 10 pixels) which have 
insufficient pixels to contribute to vascularity: this provides a reduced binary image. 

The sum of the area of the remaining Image regions in the reduced binary image is then 
10 determined at 1 0S from the results of connected component labelling, and this sum Is 
then expressed as a percentage of the area of the whole image. This procedure Is 
carried out for each of the original RGB images separately to provide three such 
percentage area values: the average of the three percentage area values is computed, 
and it represents an estimate of the percentage of the area of a tissue sample occupied 
15 by blood vessels - i.e. the sample vascularity. 

As set out in Table 10 below, vascularity is determined to be high or low depending on 
whether or not it is equal to at least 31 %. 

TABLE 10 



Desoription of vascularity 


Range 


High 


31% -100% 


Low 


0%-30% 



20 

High vascularity corresponds to relatively fast tumour growth because tumour blood 
supply has been facilitated, and early treatment is indicated. Low vascularity corresponds 
to relatively slow tumour growth, and early treatment is less important 

The procedures given in the foregoing description for calculating quantities and results 
25 can clearly be evaluated by an appropriate computer program recorded on a carrier 
medium and running on a conventional computer system. Such a program is 
straightforward for a skilled programmer to implement without requiring invention, 
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because the mathematical expressions used are well known computational procedures- 
Such a program and system will therefore not be described. 

The process steps described in the examples of all three inventions described herein are 
not all essential and alternatives may be provided. It fs for example possible to omit a 
5 step of ignoring unsuitably small areas in selecting areas for later processing, if the 
consequent increase in processing burden is acceptable. The above examples are 
intended to provide an enabling disclosure, not to limit the invention. 
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CLAIMS 



1. A method of measuring oestrogen or progesterone receptor (ER or PR) status 
having the steps of: 

a) obtaining histopathological specimen Image data; and 

b) identifying in the image data groups of contiguous pixels corresponding to 
respective eel! nuclei; 

characterised in that the method also includes the steps of: 

c) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

d) thresholding the Image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

e) determining ER or PR status from proportion of pixels corresponding to 
preferentially stained ceJIs. 

2. A method of measuring ER or PR status having the steps of; 

a) obtaining histopathological specimen Image data; and 

b) identifying in the image data groups of contiguous pixels corresponding to 
respective cell nuclei; 

characterised in that the method also includes the steps of: 

c) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

d) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

e) determining ER or PR status from normalised average saturation. 

3. A method of measuring ER or PR status having the steps oft 

a) obtaining histopathological specimen image data; and 

b) identifying in the image data groups of contiguous pixels corresponding to 
respective cell nuclei; 

characterised in that the method also includes the steps of: 

c) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 
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ci) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

s) determining ER or PR status from normalised average saturation and 
fraction of pixels corresponding to preferentially stained cells. 

4. A method according to Claim 3 characterised in that step b) is implemented using 
a K-means clustering algorithm. 

5. A method according to Claim 4 characterised in that the K-means clustering 
algorithm employs a Mahalanobis distance metric, 

6. A method according to Claim 3 characterised in that step c) Is implemented by 
transforming the image data into a chromaticity space, and deriving hue and 
saturation from image pixels and a reference colour. 

7. A method according to Claim 6 characterised in that hue is obtained from an 
angle <p equal to sin" T - = ^ and saturation from an expression 



— where (x y) and (x>y)are respectively image pixel coordinates and 
reference colour coordinates in the chromaticity space. 



8. A method according to Claim 6 characterised in that hue is adapted to lie in the 
range 0 to 90 degrees and a hue threshold of 80 degrees is set in step d). 

9. A method according to Claim 6 or 8 characterised in that a saturation threshold S a 
Is set in step d) 3 3 0 being 0.9 for saturation In the range 0.1 to 1 .9 and 0 for 
saturation outside this range. 

10. A method according to Claim 3 characterised in that the fraction of pixels 
corresponding to preferentially stained cells is determined by counting the 
number of pixels having both saturation greater than a saturation threshold and 
hue modulus less than a hue threshold and expressing suGh number as a fraction 
of a totai number of pixels in the image. 
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11. A method aocording to Claim 3 characterised in that the normalised average 
saturation is accorded a score 0, 1 , 2 or 3 according respectively to whether it is 
(i) < 25%, (il) > 25% and <S 50%, (Hi) > 50% and £ 76% or (iv) > 75% and <S 100%. 

12. A method according to Claim 11 characterised In that the fraction of pixels 
corresponding to preferentially stained ceils is accorded a score 0, 1, 2, 3, 4 or 5 
according respectively to whether it is (i) 0. (ii) > 0 and < 0.01 , (ill) s 0.01 and 
<; O.10, (iv) > 0.1 1 and < 0.33, (v) 2> 0.34 and < 0.66 or (vi) £ 0.67 and < 1.0. 

13. A method according to Claim 12 characterised In that the scores for normalised 
average saturation and fraction of pixels corresponding to preferentially stained 
cells are added together to provide a measurement of ER or PR. 

14. A method according to Claim 3 characterised in that the f raction of pixels 
corresponding to preferentially stained ceils is accorded a score 0, 1 , 2 T 3, 4 or 5 
according respectively to whether it is (i) 0. (ii) > 0 and < 0-01, (Hi) > 0.01 and 
£ 0. 1 0 f (iv) > 0. 1 1 and £ 0.33, (v) S 0,34 and < 0.66 or (vi) £ 0.67 and < 1 .0. 

15. A method according to Claim 3 characterised in that step e) te carried out by 
obtaining a score far normalised average saturation and a score for fraction Of 
pixels corresponding to preferentially stained cells and adding the scores 
together. 

16. A method according to Claim 1, 2 or 3 characterised in that rt also includes 
measuring C-erb-2 status by the following steps: 

a) correlating window functions of different lengths with pixel sub-groups 
within the Identified contiguous pixels groups to identify pixels associated 
with cell boundaries, 

b) computing brightness-related measures of oell boundary brightness and 
sharpness and brightness extent around ceil boundaries from pixels 
corresponding to cell boundaries, 

o) comparing the brightness-related measures with predetermined 
equivalents obtained from comparison images associated with different 
values of Oerb-2, and 
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d) assigning to the Image data a C-erb-2 value which is that associated with 
tha comparison image having brightness-related measures closest to 
those determined for the image data. 

17. A method according to Claim 1, 2, 3 or 16 characterised in that it also includes 
measuring vascularity by the following steps: 

a) deriving hue and saturation for the image data In a colour space having a 
hue coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the image data on the 
basis of hue and saturation; 

c) identifying in the segmented image groups of contiguous pixels; and 

d) determining vascularity from the total area of the groups of contiguous 
pixels which are sufficiently large to correspond to vascularity, such area 
being expressed as a proportion of the image data's total area. 

1 8. A method of measuring C-erb-2 status having the steps of: 

a) obtaining hfstopathological specimen image data; and 

b) identifying in the image data contiguous pixel groups corresponding to 
respective cell nuclei associated with surrounding cell boundary staining; 

characterised in that the method also includes the steps of: 

c) correlating window functions of different lengths with pixel sub-groups 
within the identified contiguous pixels groups to identify pixels associated 
with ceil boundaries, 

d) computing brightness-related measures of cell boundary brightness and 
sharpness and brightness extent around cell boundaries from pixels 
corresponding to cell boundaries, 

e) comparing the brightness-related measures with predetermined 
equivalents obtained from comparison images associated wfth different 
values of C-erb-2, and 

f) assigning to the image data a C-erb-2 value which is that associated with 
the comparison image having brightness-related measures closest to 
those determined for the image data. 
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19. A method according to Claim 1 8 characterised in that at least soma of the window 
functions have non-zero values of 6 f 12, 24 and 48 pixels respectively and zero 
values elsewhere. 



20. A method according to Claim 1 8 characterised in that pixels associated with a cell 
boundary are identified from a maximum correlation with a window function, the 
window function having a length which provides an estimate of cell boundary 
width. 



21. A method according to Claim 18 characterised in that a brightness-related 
measure of cell boundary brightness and sharpness is computed in step d) using 
a calculation including dividing cell boundaries by their respective widths to 
provide normalised boundary magnitudes, selecting a fraction of the normalised 
boundary magnitudes each greater than unselected equivalents and summing the 
normalised boundary magnitudes of the selected fraction. 

22. A method according to Claim 21 characterised In that in step d) a brightness- 
related measure of brightness extent around cell boundaries Is computed using a 
calculation including dividing normalised boundary magnitudes into different 
magnitude groups each associated with a respective range of magnitudes, 
providing a respective magnitude sum of normalised boundary magnitudes for 
each magnitude group, and subtracting a smaller magnitude sum from a larger 
magnitude sum. 

23. A method according to Claim 22 characterised in that the comparison image 
having brightness-related measures closest to those determined for the image 
data Is determined from a Euclidean distance between the brightness-related 
measures of the comparison Image and the image data. 

24. A method according to Claim 18 characterised in that in step b) identifying in the 
image data contiguous pixel groups corresponding to respective cell nuclei is 
carried out by an adaptive thresholding technique arranged to maximise the 
number of contiguous pixel groups identified. 
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25. A method according to Claim 24 wherein the image data includes red, green and 
blue image planes characterised in that the adaptive thresholding technique 
includes: 

a) generating a mean value and a standard deviation a* for pixels in the 
red image plane. 

b) generating a cyan image plane from the image data and calculating a 
mean value uc for its pixels, 

c) calculating a product CMMjic where CMM is a predetermined multiplier, 

d) calculating a quantity R a equal to the number of adjacent linear groups of 
pixels of predetermined length and including at least one cyan pixel which 
is less than CMM^c, 

e) for each red pixel calculating a threshold ' equal to {RMMur- tj R (4- Rb)} 
and RMM is a predetermined multiplier, 

f) forming a thresholded red image by discarding each red pixel that is 
greater than or equal to the threshold, 

g) determining the number of contiguous pixel groups in the thresholded red 
image, 

h) changing the values of RMM and CMM and iterating steps c) to g), 

i) changing the values of RMM and CMM once more and iterating steps c) to 

g>. 

j) comparing the numbers of contiguous pixel groups determined -in steps g} 
to IX treating the three pairs of values of RMM and CMM as points in a two 
dimensional space, selecting the pair of values of RMM and CMM 
associated with the lowest number of contiguous pixel groups, obtaining 
Its reflection in the line joining the other two pairs of values of RMM and 
CMM, using this reflection as a new pair of values of RMM and CMM and 
iterating steps c) to g) and this step J). 



26. A method according to Claim 25 characterised in that the first three pairs of RMM 
and CMM values referred to in step k) are 0.802 and 1 ,24, 0-903 and 0.903, and 
1 .24 and 0.802 respectively. 

27. A method according to Claim 25 characterised in that that it includes prior to step 
g) removing brown pixels from the thresholded red image if like-located pixels in 
the cyan Image are less than CMMji c . 
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28. A method according to Claim 25 characterised in that it includes prior to step g) 
forming an edge-filtered cyan image, generating a standard deviation <r c for its 
pixels and removing edge pixels from the thresholded red image if like-located 
pixels in the Sobel-filtered cyan image are greater than (nc+ 1-5a c )- 

29. A method according to Claim 25 characterised in that it includes prior to step g) 
removing pixels corresponding to lipids from the thresholded red image If their red 
green and blue pixel values are all greater than the sum of the relevant colour's 
minimum value and 98% of its range of pixel values in each case. 



30. A method according to Claim 25 characterised In that it includes prior to step g) 
subjecting the thresholded red image to a morphological closing operation. 

31 . A method of measuring vascularity having the steps of: 

a) obtaining histopathological specimen image data; 
characterised in that the method also includes the steps of: 

b) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

c) producing a segmented Image fay thresholding the image data on the 
basis of hue and saturation; and 

d) identifying in the segmented image groups of contiguous pixels; and 

e) determining vascularity from the total area of the groups of contiguous 
pixels which are sufficiently large to correspond to vascularity, such area 
being expressed as a proportion of the image data's total area. 

32. A method according to Claim 31 wherein the image data comprises pixels with 
red, green and blue values designated R T G and B respectively, characterised in 
that a respective saturation value S is derived in step b) for each pixel by: 

i) defining M and m for each pixel as respectively the maximum and minimum 
of R, G and B; and 

ii) setting S to zero if m equals zero and setting S to (M - m)/M otherwise. 

33. A method according to Claim 32 characterised in that hue values designated H 
are derived by: 
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a) defining new values newr, newg and newb for each pixel given by newr = 
(M - R)/(M - m), newg = (M- Q)/(M - m) and newb = (M - B)/(M - m) in 
order to convert each pixel value rnto the difference between Its magnitude 
and that erf the maximum of the three colour magnitudes of that pixel, this 
difference being divided by the difference between the maximum and 
minimum of R, Q and B. and 

b) calculating H as tabulated immediately below: 



M 


H 


0 


180 


R 


60(newb - newg)* 


G 


60(2 + newr -newb)* 


B 


60(4 + newg - newr)* 



* provided that if H proves to be >360, then 360 is subtracted from it T and 
If H proves to be <0, 360 is added to it. 

34. A method according to Claim 33 characterised in that the step of producing a 
segmented image is implemented by designating for further processing only 
those pixels having both a hue H in the range 282-356 and a saturation S in the 
range 0.2 to 0.24. 



35. A method according to Claim 34 characterised in that the step of identifying in the 
segmented image groups of contiguous pixels includes the step of spatially 
filtering such groups to remove groups having insufficient pixels to contribute to 
vascularity. 



36. A method according to Claim 35 characterised in that the step of determining 
vascularity includes treating vascularity as having a high or a low value according 
to whether or not it is at least 31 %. 

37. A computer program for measuring ER or PR status, the program being arranged 
to control computer apparatus to execute the steps of: 
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a) processing histopathological specimen image data to identify in the Image 
data groups of contiguous pixels corresponding to respective cell nuclei; 

characterised In that the program Is also arranged to implement the steps of: 

b) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

c) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

d) determining ER or PR status from proportion of pixels corresponding to 
preferentially stained cells. 

38. A computer program for measuring ER or PR status, the program being arranged 
to control computer apparatus to execute the steps of: 

a) processing histopathological specimen image data to identify in the image 
data groups of contiguous pixels corresponding to respective cell nuclei; 

characterised in that the program is also arranged to implement the steps of: 

b) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

o) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

d) determining ER or PR status from normalised average saturation, 

39. A computer program for measuring ER or PR status, the program being arranged 
to control computer apparatus to execute the steps of: 

a) processing histopathological specimen image data to identify in the image 
data groups of contiguous pixels corresponding to respective cell nuclei; 

characterised in that the program is also arranged to implement the steps of: 

b) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

c) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

d) determining ER or PR status from normalised average saturation and 
fraction of pixels corresponding to preferentially stained ceils. 
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40. A computer program according to Claim 39 characterised tn that step a) is 
implemented using a K-means clustering algorithm. 

41. A computer program according to Claim 39 characterised in that step b) is 
implemented by transforming the image data into a chromatidty space, and 
deriving hue and saturation from image pixels and a reference colour, 

42. A computer program according to Claim 41 characterised in that hue is obtained 



from an angle 0 equal to sin" 1 , i ■ saturation from an 

expression — — £|- B where y) and (jf r y)are respectively image pixel 
x + y 

coordinates and reference colour coordinates in the chromaticity space. 

43, A computer program according to Claim 41 characterised in that hue is adapted 
to lie in the range 0 to 90 degrees and a hue threshold of 80 degrees is set In 
step c). 

44. A computer program according to Claim 41 characterised in that a saturation 
threshold S 0 is set in step c), S 0 being 0.9 for saturation in the range 0.1 to 1,9 
and 0 for saturation outside this range. 

.45. A computer program according to Claim 39 characterised In that the fraction of 
pixels corresponding to preferentially stained cells is determined by counting the 
number of pixels having both saturation greater than a saturation threshold and 
hue modulus less than a hue threshold and expressing such number a© a fraction 
of a total number of pixels in the image. 

46. A computer program according to Claim 39 characterised In . that the normalised 
average saturation is accorded a score 0, 1, 2 or 3 according respectively to 
whether it Is (i) < 25%, (ii) > 25% and £ 50%, (iii) > 50% and < 75% or (iv) > 75% 
and £100%. 

47. A computer program according to Claim 46 characterised in that the fraction of 
pixels corresponding to preferentially stained cells is accorded a score 0, 1 , 2, 3, 
4 or 5 according respectively to whether it is (i) 0, (ii) > 0 and < 0.01 f (Hi) > Q.01 
and £ 0.1 0, (iv) > 0.1 1 and < 0.33, (v) > 0.34 and < 0.66 or (vi) S: 0.67 and <: 1 .0. 
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48. A computer program according to Claim 47 characterised In that the scores for 
normalised average saturation and fraction of pfacals corresponding to 
preferentially stained cells are added together to provide a measurement of ER or 
PR. 

49. A computer program according to Claim 39 characterised in that the fraction of 
pixels corresponding to preferentially stained cells is accorded a score 0, 1 , 2, 3 a 
4 or 5 according respectively to whether it is (i) 0, (ii) > 0 and < 0.01 , (Hi) > 0.01 
and < 0.10, (lv> 2> 0,1 1 and < 0.33, (v) > 0.34 and < 0.66 or (vi) > 0.67 and < 1 .0. 

50. A computer program according to Claim 39 characterised in that step e) is carried 
out by obtaining a score for normalised average saturation and a score for 
fraction of pixels corresponding to preferentially stained cells and adding the 
scores together. 

51. A computer program according to Claim 37, 38 or 39 characterised in that it is 
also arranged for derivation of a measure C-erb-2 status by: 

a) correlating window functions of different lengths with pixel sub-groups 
within the identified contiguous pixels groups to identify pixels associated 
with cell boundaries, 

b) computing brightness-related measures of cell boundary brightness and 
sharpness and brightness extent around cell boundaries from pixels 
corresponding to cell boundaries, 

c) comparing the brightness-related measures with predetermined 
equivalents obtained from comparison images associated with different 
values of C-erb-2, arid 

d) assigning to the image data a C-erb-2 value which is that associated with 
the comparison Image having brightness-related measures closest to 
those determined for the image data. 



52. A computer program according to Claim 37. 38, 39 or 61 characterised in that it is 
also arranged for derivation of a measure C-ert>2 status by: 

a) deriving hue and saturation for the image data in a colour Space having a 
hue coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the image data on the 
basis of hue and saturation; and 
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c) identifying in the segmented Image groups of contiguous pixels; and 

d) determining vascularity from the total area of the groups of contiguous 
pixels which are sufficiently lange to correspond to vascularity, such area 
being expressed as a proportion of the image data's total area. 



53. A computer program for use in measuring C-erb-2 status arranged to control . 
computer apparatus to execute the steps of: 

a) processing histopathologic^ specimen image data to identify contiguous 
pixel groups corresponding to respective cell nuclei associated with 
surrounding cell boundary staining; 

characterised in that the computer program is also arranged to implement the 
steps of: 

b) correlating window functions of different lengths with pixel sub-groups 
within the identified contiguous pixels groups to identify pixels associated 
with cell boundaries, 

c) computing brightness-related measures of cell boundary brightness and 
sharpness and brightness Bxtent around cell boundaries from pixels 
corresponding to cell boundaries, 

d) comparing the brightness-related measures with predetermined 
equivalents obtained from comparison images associated with different 
values of C-erb-2, and 

e) assigning to the image data a C-erb-2 value which Is that associated with 
the comparison image having brightness-related measures closest to 
those determined for the image data. 

54. A computer program according to Claim 53 characterised in that at least some of 
the window functions have non-zero values of 6, 12, 24 and 48 pixels respectively 
and zero values elsewhere, 

55. A computer program according to Claim 53 characterised in that pixels 
associated with a cell boundary are identified from a maximum correlation with a 
window function, the window function having a length which provides an estimate 
of cell boundary width. 
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56. A computer program according to Claim 53 characterised in that in step d) a 
brightness-related measure of cell boundary brightness and sharpness is 
computed using a calculation including dividing cell boundaries by their respective 
widths to provide normalised boundary magnitudes, selecting a fraction of the 
normalised boundary magnitudes each greater than undetected equivalents and 
summing the normalised boundary magnitudes of the selected fraction. 

57. A computer program according to Claim 53 characterised in that in step d) a 
brightness-related measure of brightness extent around cell boundaries Is 
computed using a calculation including dividing normalised boundary magnitudes 
into different magnitude groups each associated with a respective rang© of 
magnitudes, providing a respective magnitude sum of normalised boundary 
magnitudes for each magnitude group, and subtracting a smaller magnitude sum 
from a larger magnitude sum, 

58. A computer program according to Claim 57 characterised in that the comparison 
image having brightness-related measures closest to those determined for the 
image data is determined from a Euclidean distance between the brightness- 
related measures of the comparison image and the image data. 

59. A computer program according to Claim 53 characterised in that in step b) 
identifying in the image data contiguous pixel groups corresponding to respective 
cell nuclei is carried out by an adaptive thresholding technique arranged to 
maximise the number of contiguous pixel groups Identified. 

60. A computer program aocording to Claim 59 wherein the image data includes red r 
green and blue image planes characterised In that the adaptive thresholding 
technique includes: 

a) generating a mean value and a standard deviation oft for pixels in the 
red image plane, 

b) generating a oyan image plane from the imaga data and calculating a 
mean value \xc for Its pixels, 

c) calculating a product CMMfic where CMM is a predetermined multiplier, 
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d) calculating a quantity R B equal to the number of adjacent linear groups of 
pixels of predetermined length and including at least one cyan pixel which 
Is less than CMMjic, 

e) for each red pixel calculating a threshold equal to {RMMjar- ctr(4 - R 3 )} 
and RMM is a predetermined multiplier, 

f) forming a threshoided red image by discarding each red pixel that is 
greater than or equal to the threshold, 

g) determining the number of contiguous pixel groups in the threshoided red 
image, 

h) changing the values of RMM and CMM and iterating steps c) to g), 

i) changing the values of RMM and CMM once more and iterating steps c) to 

g). 

j) comparing the numbers of contiguous pbce! groups determined in steps g) 
to i), treating the three pairs of values of RMM and CMM as points in a two 
dimensional space, selecting the pair of values of RMM and CMM 
associated with the lowest number of contiguous pixel groups, obtaining 
its reflection in the line joining the other two pairs of values of RMM and 
CMM, using this reflection as a new pair of values of RMM and CMM and 
iterating steps o) to g) and this step j). 

61 . A computer program according to Claim 60 characterised in that the first three 
pairs of RMM and CMM values referred to in step k) are 0.S02 and 1.24, 0.903 
and 0.903, and 1.24 and 0.802 respectively. 

62. A oomputer program according to Claim 60 characterised In that that the adaptive 
thresholding technique includes prior to step g) removing brown pixels from thB 
threshoided red image if like-located pixels in the cyan image are less than 
CMMnc- 

63. A computer program according to Claim 60 characterised in that the adaptive 
thresholding technique includes prior to step g) forming an edge-filtered cyan 

image, generating a standard deviation er c for Its pixels and removing edge pixels 
from the threshoided red image K tike-located pixels in the Sobel-fittered cyan 

image are greater than (|i c + 1.5cc). 



25-SEP-2002 12=41 



25-SEP-2002 12:41 FROM IP MALUERN TO UK PATENT P. 24 




54 

64. A computer program according to Claim 60 characterised In that the adaptive 
thresholding technique Includes prior to step g) removing pixels corresponding to 
lipids from the thresholded red Image if their red green and blue pixel values are 
all greater than the sum of the relevant colour's minimum value and 98% of its 
range of pixel values in each case. 

65. A computer program according to Claim 60 characterised in that the adaptive 
thresholding technique includes prior to step g) subjecting the thresholded red 
image to a morphological closing operation. 

66. A computer program tor use in measuring vascularity characterised In that it is 
arranged to control computer apparatus to execute the steps of: 

a) using histopathologic^ specimen image data to derive hue and saturation 
for the image data in a colour space having a hue coordinate and a 
saturation coordinate; 

b) producing a segmented image by thresholding the image data on the 
basis of hue and saturation; and 

c) identifying in the segmented image groups of contiguous pixels; and 

d) determining vascularity from the total area of the groups of contiguous 
pfxels which are sufficiently large to correspond to vascularity, such area 
being expressed as a proportion of the image data's total area. 

67. A computer program according to Claim 66 wherein the image data comprises 
pixels with red, green and blue values designated R, © and B respectively, 
characterised in that a respective saturation value S is derived in step b) for each 
pixel by: 

i) defining M and m for each pixel as respectively the maximum and minimum 
of R, Q and B; and 

ii) setting S to zero ft m equals zero and setting S to (M - m)/M otherwise* 

66. A computer program according to Claim 67 characterised in that hue values 
designated H are derived by: 

a) defining new values newr, newg and newb for each pixel given by newr = 
(M - R)/(M -m), newg = (M - G)/(M - m) and newb = (M - B)/(M - m) in 
order to convert each pixel value into the difference between its magnitude 
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and that of the maximum of the three colour magnitudes of that pixel, this 
difference being divided by the difference between the maximum and 
minimum of R, G and B, and 

b) calculating H as tabulated Immediately below: 



M 


H 


0 


180 


R 


60{newb - newg)* 


G 


60(2 + newr- newb)* 


B 


60(4 .+ newg - newr)* 



* provided that if H proves to be >360 B then 360 is subtracted from rt a and 
if H proves to be <0, 360 is added to it. 



69. A computer program according to Claim 68 characterised in that the step of 
producing a segmented image is implemented by designating for further 
processing only those pixels having both a hue H in the range 282-356 and a 
saturation S in the range 0.2 to 0.24. 

70. A computer program according to Claim 69 characterised in that the step of 
identifying in the segmented image groups of contiguous pixels Includes the step 
of spatially filtering such groups to remove groups having insufficient pixels to 
contribute to vascularity. 



71. A computer program according to Claim 70 characterised in that the step of 
determining vascularity includes treating vascularity as having a high or a low 
value according to whether or not it is at least 31 %, 



72. Apparatus for measuring ER or PR status including means for photographing 
histopathoiogical specimens to provide image data and computer apparatus to 
process the image data, the computer apparatus being programmed to identify in 
the image data groups of contiguous pixels corresponding to respective call 
nuclei, characterised in that the computer apparatus is also programmed to 
execute the steps of: 
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a) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

b) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

o) determining ER or PR status from proportion of pixels corresponding to 
preferentially stained cells. 

73. Apparatus for measuring ER or PR status including means for photographing 
histopathologic^ specimens to provide image data and computer apparatus to 
process the image data, the computer apparatus being programmed to identify In 
the image data groups of contiguous pixels corresponding to respective eel! 
nuclei, characterised in that the computer apparatus is also programmed to 
execute the steps of: 

a) deriving hue and saturation for the image data in a colour space having a 

hue coordinate and a saturation coordinate; 

i 

b) thresholding the intege data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

c) determining ER or PR status from normalised average saturation. 

74, Apparatus for measuring ER or PR status Including means for photographing 
htstopathological specimens to provide image data and computer apparatus to 
process the image data, the oomputer apparatus being programmed to identify in 
the image data groups of contiguous pixels corresponding to respective ceil ' 
nuclei, characterised in that the computer apparatus is also programmed to 
execute the steps of: 

a) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

b) thresholding the image data on the basis of hue and saturation and 
identifying pixels corresponding to cells which are preferentially stained 
relative to surrounding specimen tissue; and 

c) determining ER or PR status from normalised average saturation and 
f raotion of pixels corresponding to preferentially stained cells. 
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75. Apparatus according to Claim 74 characterised in that step a) is Implemented by 
transforming the image data into a chromaticity space, and deriving hue and 
saturation from image pixels and a reference colour- 

76. Apparatus according to Claim 75 characterised in that hue is obtained from an 



w _ ~| 

angle $ equal to sin' 1 ... = ^ ' and saturation from an expression 

*f , where and (x, 5?) are respectively image pixel coordinates and 
x + y 

reference colour coordinates in the chromaticity space. 

77. Apparatus according to Claim 76 characterised in that hue is adapted to lie in the 
range 0 to 90 degrees and a hue threshold of 80 degrees is set In step b). 

78. Apparatus according to Claim 74 characterised In that a saturation threshold 3 Q is 
set in step b), S a being 0.9 for saturation in the range 0.1 to 1 .9 and 0 for 
saturation outside this range. 

79. Apparatus according to Claim 74 characterised in that the fraction of pixels 
corresponding to preferentially stained cells is determined by counting the 
number of pixels having both saturation greater than a saturation threshold and 
hue modulus less than a hue threshold and expressing such number as a fraction 
of a total number of pixels in the image. 

SO, Apparatus according to Glafm 74 characterised in that the normalised average 
saturation is accorded a score 0, 1 , 2 or 3 according respectively to whether It is 
(i) < 25%, (ii) > 25% and £ 50%, (ill) > 50% and < 75% or (iv) > 75% and S 100%, 

81. Apparatus according to Claim 80 characterised in that the fraction of pixels 
corresponding to preferentially stained cells Is accorded a score 0, 1 , 2, 3, 4 or 5 
according respectively to whether it is <i) 0 T (ii) > 0 and < 0.01 , (iii) > 0.01 and 
<S 0.10, (iv) > 0.1 1 and ^ 0.33, (v) £ 0>34 and < 0.66 or (vi) 5> 0.67 and < 1.0. 

82. Apparatus according to Claim 81 characterised in that the scores for normalised 
average saturation and fraction of pixels corresponding to preferentially stained 
ceils are added together to provide a measurement of ER or PR- 
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S3* Apparatus according to Claim 74 characterised in that the fraction of pixels 
corresponding to preferentially stained cells is accorded a score 0, 1 , 2, 3, 4 or 5 
according respectively to whether it is (i) 0. (ii) > 0 and < 0,01 , (Hi) > 0.01 and 
£ 0.10, (iv) * OA 1 and < 0.33, (v) > 0.34 and £ 0.66 or (vi) > 0.67 and < 1 .0. 

84, Apparatus according to Claim 74 characterised in that step c) is carried out by 
obtaining a score for normalised average saturation arid a score for fraction of 
pixels corresponding to preferentially stained cells and adding the scores 
together. 

85. Apparatus according to Claim 72, 73 or 74 characterised In that it is also arranged 
to determine C-erb-2 status and the computer apparatus is also programmed to: 

a) correlate window functions of different lengths with pixel sub-groups within 
the identified contiguous pixels groups to identify pixels associated with 
ceil boundaries, 

b) compute brightness-related measures of cell boundaiy brightness and 
sharpness and brightness extent around ceil boundaries from pixels 
corresponding to cell boundaries, 

c) compare the brightness-related measures with predetermined equivalents 
obtained from comparison images associated with different values of 
C-erb-2 t and 

d) assign to the image data a C-erb-2 value which Is that associated with the 
comparison image having brightness-related measures closest to those 
determined for the image data. 

86* Apparatus according to Claim 72 t 73, 74 or 85 characterised in that it is also 
arranged to determine vascularity and the computer apparatus is also 
programmed to: 

a) derive hue and saturation for the image data In a colour space having a 
hue coordinate and a saturation coordinate; 

b) produce a segmented image by thresholding the image data on the basis 
of hue and saturation; 

c) identify in the segmented image groups of contiguous pixels; and 
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d) determine vascularity from the total area of the groups of contiguous 
pixels which are sufficiently large to correspond to vascularity, such area 
being e^ressed as a proportion of the Image date's total area. 

87. Apparatus for measuring C-erb-2 status including means for photographing 
histopathoJogical specimens to provide Image data and computer apparatus to 
process the image data, the computer apparatus being programmed to identify in 
the image data groups of contiguous pixels corresponding to respective cell 
nuclei, characterised in that the computer apparatus is also programmed to 
execute the steps of: 

a) correlating , window functions of different lengths with pfocef sub-groups 
within the identified contiguous pixels groups to identity pixels associated 
with cell boundaries, . 

b) computing brightness-related measures of cell boundary brightness and 
sharpness and brightness extent around cell boundaries from pixels 
corresponding to cell boundaries, 

c) comparing the brightness-related measures with predetermined 
equivalents obtained from comparison Images associated with different 
values of C-erb-2, and 

d) assigning to the image data a Oerb-2 value which is that associated with 
the comparison image having brightness-related measures closest to 
those determined for .the image data. 

B8. Apparatus according to Claim 87 characterised in that at least some of the 
window functions have non-zero values of 6, 12, 24 and 48 pixels respectively 
and zero values elsewhere. 

89. Apparatus according to Claim 87 characterised in that the computer apparatus is 
programmed to identify pixels associated with a call boundary from a maximum 
correlation with a window function, the window function having a length which 
provides an estimate of cell boundary width. 

90. Apparatus according to Claim 87 characterised in that the computer apparatus is 
programmed to execute step b) by computing a brightness-related measure of 
cell boundary brightness and sharpness using a calculation including dividing ceil 
boundaries by their respective widths to provide normalised boundary 
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magnitudes, selecting a fraction of the normalised boundary magnitudes each 
greater than unselected equivalents and summing the normalised boundary 
magnitudes of the selected fraction. 



91. Apparatus according to Claim B7 characterised in that the computer apparatus Is 
programmed to execute step b) by computing a brightness-related measure of 
brightness extent around cell boundaries using a calculation including dividing 
normalised boundary magnitudes Into different magnitude groups each 
associated with a respective range of magnitudes, providing a respective 
magnitude sum of normalised boundary magnitudes for each magnitude group, 
and subtracting a smaller magnitude sum from a larger magnitude sum. 

92. Apparatus according to Claim 91 characterised In that the computer apparatus is 
programmed to determine the comparison image having brightness-related 
measures closest to those determined for the image data from a Euclidean 
distance between the brightness-related measures of the comparison Image and 
the image data. 

93. Apparatus according to Claim 87 charactBrised in that the computer apparatus is 
programmed to identify in the image data contiguous pixel groups corresponding 
to respective cell nuclei by an adaptive thresholding technique arranged to 
maximise the number of contiguous pixel groups Identified. 

94. Apparatus according to Claim 93 wherein the image data includes red, green and 
blue Image planes characterised in that the adaptive thresholding technique 
includes: 

a) generating a mean value pa and a standard deviation a R for pixels in the 
red image plane, 

b) generating a cyan image plane from the Image data and calculating a 
mean value ncfor its pixels, 

c) calculating a product CWIMpo where CMM is a predetermined multiplier, 

d) calculating a quantity R H equal to the number of adjacent linear groups of 
pixels of predetermined length and including at least one cyan pixel which 
is less than CMM^ic, 



25-5EP-2002 12:43 FROM IP MRLUERN 



TO UK PATENT 



P. 31 



e) for each red pixel calculating a threshold equal to {RMMjah- o h (4- R b )} 
and RMM is a predetermined multiplier, 

f) forming a thresholded fad image by discarding each red pixel that is 
greater than or equal to the threshold, 

g) determining the number of contiguous pixel groups in the thresholded red 

image, 

h) changing the values of RMM and CMM and iterating steps c) to g), 

i) changing the values of RMM and CMM once more and iterating steps c) to 
9)3 

j) comparing the numbers of contiguous pixel groups determined in steps g) 
to i), treating the three pairs of values of RMM and CMM as points in a two 
dimensional space, selecting the pair of values of RMM and CMM 
associated with the lowest number of contiguous pixel groups, obtaining 
its reflection In the line joining the other two pairs of values of RMM and 
CMM, using this reflection as a new pair of values of RMM and CMM and 
iterating steps c) to g) and this step j). 



95. Apparatus according to Claim 94 characterised in that the first three pairs of RMM 
and CMM values referred to in step k) are 0.802 and 1.24, 0,903 and 0.903, and 
1.24 and 0,802 respectively. 

96. Apparatus according to Claim 94 characterised in that the computer apparatus is 
programmed to remove brown pixels from the thresholded red image prior to step 
g) if like-located pixels In the cyan image are less than CMMjac. 

97. Apparatus according to Claim 94 characterised in that the computer apparatus is 
programmed to form an edge^tiltered cyan image, generate a standard deviation 
oc for its pixels and remove edge pixels from the thresholded red image prior to 
step g) if like-located pixels? in the Sobel-filtered cyan image are greater than 
fti c +1.5ac). 

98. Apparatus according to Claim 94 characterised in that the computer apparatus is 
programmed to remove pixels corresponding to lipids from the thresholded red 
image prior to step g) if their red green and blue pixel values are all greater than 
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the sum of the relevant colour's minimum value and 98% of its range of pixel 
values in each case. 

99. Apparatus according to Claim 94 characterised in that the computer apparatus Is 
programmed to subject the thresholded red image to a morphological closing 
operation prior to step g). 

100. Apparatus for measuring vascularity Including means for photographing 
histopathological specimens to provide image data and computer apparatus to 
process the image data, characterised in that the computer apparatus is also 
programmed to execute the steps of: 

a) deriving hue and saturation for the image data in a colour space having a 
hue coordinate and a saturation coordinate; 

b) producing a segmented image by thresholding the Image data on the 
basis of hue and saturation; and 

c) identifying in the segmented image groups of contiguous pixels; and 

d) determining vascularity from the total area of the groups of contiguous 
pixels which are sufficiently large to correspond to vascularity, such area 
being expressed as a proportion of the image data's total area. 

101. Apparatus according to Claim 100 wherein the image data comprises pixels with 
red, green and blue values designated R„ G and B respectively, characterised in 
that the computer apparatus Is programmed to derive a respective saturation 
value S for each pixel in step b) by: 

i) defining M and m for eaoh pixel as respectively the maximum and minimum 
of R, G and B; and 

Ii) setting S to zero if m equals zero and setting S to (M - rnVM otherwise. 

1 02. Apparatus according to Claim 101 characterised In that the computer apparatus is 
programmed to derive hue values designated H by: 

a) defining new values newr, newg and newb for each pixel given by newr = 
(M - R)/(M - m), newg = (M - G)/(M - rn) and newb = (M - B)/(M - m) in 
order to convert eaoh pixel value into the difference between its magnitude 
and that of the maximum of the three colour magnitudes of that pixel, this 



25-5EP-2002 125 44 FROM IP MRLUERN 



TO UK POTENT 



P. 33 



63 



b) 



difference being divided by the difference between the maximum and 

minimum of R, Q and B r and 

calculating H as tabulated Immediately below: 



M 


H 


0 


180 


R 


60(newb-newg)* 


G 


60(2 + newr - newb)" 


B 


60(4 + newg-newr)* 



* provided that if H proves to be >360, then 360 is subtracted from it, and 
if H proves to be <0, 360 is added to it. 

103. Apparatus according to Claim 102 characterised in that the computer apparatus is 
programmed to produce a segmented image by designating for further processing 
only those pixels having both a hue H in the range 282-355 and a saturation S in 
the range 0.2 to 0.24. 

1 04. Apparatus according to CJaim 1 03 characterised In that the computer apparatus is 
programmed to identify In the segmented image groups of contiguous pixels by 
spatially filtering such groups to remove groups having insufficient pixels to 
contribute to vascularity. 

1 05. Apparatus according to Claim 1 00 characterised in that the computer apparatus is 
programmed to determine vascularity by treating it as having a high or a low 
value according to whether or not it is at least 31%. 
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ABSTRACT 

A method of measuring oestrogen or progesterone receptor (ER or PR) comprises 
identifying in histopathologioal specimen Image data pixel groups Indicating cell nuclei, 

5 and deriving image hue and saturation. The image is thresholded using hue and 
saturation and preferentially stained cells identified. ER or PR status Is determined from 
normalised average saturation and proportion of preferentially stained cells. A method of 
measuring C-erb-2 comprises correlating window functions with pixel sub-groups to 
identify ceil boundaries, computing measures of cell boundary brightness and sharpness 

10 and brightness extent around cell boundaries, and comparing the measures with 
comparison images associated with different values of Oerb-2. A C-erb-2 value 
associated with a comparison image having similar brightness-related measures is 
assigned- A method of measuring vascularity comprises deriving image hue and 
saturation, producing a segmented image by hue and saturation thresholding and 

15 identifying contiguous pixels. Vascularity is determined from contiguous pixel area 
corresponding to vascularity expressed as a proportion of total image area. 
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Figure 2 ER & PR measurement j 
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Figure 4, Colour cube to chromacity transformation 
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Figure 5 Chromaticity Space Reference System 
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Figure 6 Polar coordinate system 
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Figure 7, Cerb-B2 Process 
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